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Executive  Summary 


This  study  provides  baseline  measures  on  the  Automated  Radar  Terminal  System  (ARTS)  IHA. 
The  Federal  Aviation  Administration  had  previously  identified  six  high-level  operational 
constructs  to  be  used  in  the  assessment  of  en  route  air  traffic  control  systems:  Safety,  Capacity, 
Performance,  Workload,  Usability,  and  Simulation  Fidelity.  Engineering  research  psychologists 
from  the  Human  Factors  Branch  (ACT-530)  adapted  these  constructs  to  the  terminal  domain  and 
based  this  assessment  of  the  ARTS  IILA  on  them. 

The  researchers  created  two  simulation  scenarios  of  Boston  Terminal  Radar  Approach  Control 
(TRACON)  airspace.  The  air  traffic  patterns  and  airspace  characteristics  of  these  scenarios  were 
representative  of  four  sectors  at  Boston  TRACON  and  used  two  runway  configurations.  Each 
scenario  used  a  90*  percentile  day  for  traffic  volume.  The  four  simulated  sectors  were  Initial 
Departure,  South,  Rockport,  and  Final  One. 

The  Target  Generation  Facility  and  ARTS  IDA  Continuous  Data  Recording  tapes  provided 
objective  measures  of  controller  and  system  performance.  Controller  and  expert  observer 
questionnaires  provided  subjective  data.  This  study  contains  statistics  at  several  levels  of 
specificity:  across  the  four  sectors,  by  individual  sectors,  and  by  15-min  intervals. 

This  report  presents  guidance  on  using  the  baseline  measures  to  verify  the  effectiveness  and 
efficiency  of  a  future  terminal  air  traffic  control  system.  It  recommends  a  process  to  merge 
quantitative  statistics  with  controller  expert  opinion  in  order  to  compare  the  baseline  and  future 
systems.  The  data  reported  here  should  only  be  used  for  these  purposes. 


1 .  Introduction 


As  it  moves  into  the  21st  century,  the  Federal  Aviation  Administration  (FAA)  will  specify, 
prototype,  develop,  test,  and  deploy  new  air  traffic  control  (ATC)  automation  systems  for  the 
terminal  domain.  These  systems  will  replace  or  augment  systems  currently  in  use.  This  report 
provides  baseline  data  on  the  efficiency  and  effectiveness  of  the  Automated  Radar  Terminal 
System  (ARTS)  niA  that  may  be  useful  throughout  this  process. 

1.1  Background 

As  part  of  an  earlier  effort  to  provide  baseline  data  for  the  current  en  route  system,  the  Air  Traffic 
Advanced  Automation  System  Requirements  Organization,  ATR-320,  identified  six  high-level 
operational  constructs  useful  for  system  comparisons,  as  follows: 

a.  Safety  represented  the  extent  to  which  the  system  maintmned,  enhanced,  or  degraded 
relative  safety. 

b.  Capacity  measured  aspects  of  traffic  throughput  in  a  specific  sector  of  airspace  during  a 
specified  time. 

c.  Performance  involved  controller  interaction  with  the  system  through  the  computer-human 
interface  (CHI). 

d.  Workload  represented  subjective  evaluations  of  cognitive  task  demands  of  ATC 
simulations. 

e.  Usability  consisted  of  user  opinions  regarding  the  acceptability  of  the  CHI,  controls, 
displays,  and  other  equipment  items. 

f.  Simulation  Fidelity  represented  characteristics  of  the  air  traffic  mix  and  the  perceived 
fidelity  of  the  simulation  scenarios. 

Galushka,  Frederick,  Mogford,  and  Krois  (1995)  developed  a  set  of  baseline  measures  based  on 
these  constructs  through  meetings  with  en  route  controllers.  During  these  sessions,  they 
reviewed  all  available  metrics  and  identified  a  set  that  was  useful  for  system  comparisons.  They 
based  some  variables  in  their  original  set  on  work  by  Buckley,  DeBaryshe,  Hitchner,  and  Kohn 
(1983);  Hedge,  Borman,  Hanson,  Carter,  and  Nelson  (1993);  and  Sollenberger,  Stein,  and 
Gromelski  (1997).  These  variables  served  as  the  basis  for  the  Plan  View  Display  Baseline  (PVD) 
study  conducted  in  1995. 

1 .2  Purpose 

The  goal  of  the  current  study  was  to  identify  and  collect  baseline  measures  that  would  be 
effective  indicators  of  ARTS  niA  performance  and  suitable  for  comparisons  with  future  terminal 
ATC  automation  systems.  To  accomplish  this  goal,  engineering  research  psychologists  from  the 
FAA  William  J.  Hughes  Technical  Center,  Human  Factors  Branch,  ACT-530,  and  personnel 
from  Boston  Terminal  Radar  Approach  Control  (TRACON)  reviewed  the  measures  used  in  the 
en  route  baseline  to  assess  their  applicability  to  the  terminal  domain.  They  refined  several 
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measures  and  added  new  measures  specific  to  terminal  ATC  operations.  The  final  set  of  terminal 
baseline  measures  contained  both  objective  and  subjective  elements. 

Objective  measures  were  quantitative  metrics  that  were  pertinent  to  the  ATC  mission  and 
realistic  concerning  ATC  operations.  Subjective  measures  were  controller  and  expert  observer 
opinions  and  perceptions  collected  from  questionnaires  and  rating  scales. 

The  measures  collected  during  the  terminal  baseline  simulations  provide  indices  of  relative  levels 
of  operational  acceptability  and  cannot  be  used  in  isolation.  Variations  between  the  ARTS  TTTA 
and  other  systems  on  the  reported  variables  must  be  analyzed  in  the  context  advised  in  this 
document  to  derive  valid  system  comparisons.  Any  other  use  of  these  data  might  prove 
misleading  and  invalid. 

2.  Method 


This  study  involved  Full  Performance  Level  terminal  controllers  working  four  simulated  sectors 
of  Boston  TRACON  airspace.  A  variety  of  data  sources  provided  objective  and  subjective 
measures  of  controller  and  system  performance.  These  measures  followed  the  six  high-level 
operational  constructs  identified  by  ATR-320. 

Three  engineering  research  psychologists  and  a  data  collection  specialist  managed  the  activity 
and  collected  objective  and  subjective  data.  Specialists  from  the  TGF  and  the  ARTS  TTIA 
Laboratory  provided  simulation  hardware  and  software  support. 

2. 1  Participants 

Twelve  Boston  TRACON  controllers  participated  in  groups  of  four,  one  group  per  week,  for 
3  consecutive  weeks.  The  average  age  of  the  controllers  was  34.0  (SD  =  3.87)  years  with  an 
average  of  12.3  (SD  =  2.93)  years  of  experience  controlling  traffic  and  an  average  of  6.7  (SD  = 
3.70)  years  of  experience  with  the  ARTS  IHA.  The  controllers  were  current  and  knowledgeable 
on  the  four  sectors  used  in  this  study. 

Three  Boston  TRACON  supervisors  served  as  expert  observers,  one  per  week.  They  assisted 
with  data  collection  and  made  performance  evaluations.  Their  primary  responsibilities  were  to 
complete  the  Observer  Evaluation  Form  (Appendix  A)  and  to  provide  procedural  and  operational 
expertise  when  necessary. 

Sixteen  Simulation  Operation  Pilots  (SIMOPs)  from  the  Technical  Center  Target  Generation 
Facility  (TGF)  controlled  simulated  aircraft  targets.  The  SIMOPs  provided  voice 
communications  and  made  heading,  altitude,  and  speed  changes  using  special  computer 
workstations.  Most  SIMOPs  were  not  professional  pilots  but  had  training  in  aviation 
terminology,  were  familiar  with  ATC  procedures,  and  had  received  training  on  the  Boston 
TRACON  airspace. 
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2.2  Boston  Terminal  Radar  Approach  Control  Airspace 


An  imaginary  and  approximate  line  in  space  defines  the  Boston  TRACON  airspace.  It  begins 
over  Providence,  RI;  bears  north  to  Gardner,  MA;  then  east  to  Plum  Island,  MA.  This  boundary 
line  continues  southeast  to  a  point  25  nmi  east  of  Boston  (SCUPP  Intersection);  southwest  to 
Plymouth,  MA;  and  west  to  Providence.  The  airspace  begins  at  the  surface  and  extends 
vertically  to  14,000  ft.  Many  areas  (called  shelves),  where  altitudes  of  control  can  vary  based  on 
sector,  are  found  along  the  outer  edges  of  the  airspace.  The  Boston  TRACON  controls  all 
Instrument  Flight  Rules  (HTl)  and  Visual  Flight  Rules  (VFR)  traffic  that  either  originates, 
terminates,  or  transits  through  the  airspace.  Boston  TRACON  has  responsibility  for  the  Logan 
International  Airport  (BOS)  and  many  satellite  airports  in  the  metropolitan  Boston  area. 

At  the  heart  of  the  Boston  TRACON  airspace  is  the  Boston  Class  B  airspace.  The  airspace  is 
centered  at  BOS  and  the  Boston  Very  High  Frequency  Omnidirectional  Radio  Range  (VOR)  and 
it  extends  approximately  20  nmi  in  all  directions.  The  altitude  floor  of  Class  B  airspace  varies 
depending  on  the  distance  from  BOS,  and  the  altitude  ceiling  is  7000  ft  in  all  areas.  The  purpose 
of  Class  B  airspace  is  to  prevent  collisions  between  VFR  aircraft  operating  in  proximity  to  BOS 
and  high-performance  turboprop  or  jet  aircraft  also  using  the  airport. 

Boston  TRACON  is  composed  of  eight  sectors.  The  amount  of  airspace  controlled  by  a  sector 
can  vary  based  on  workload,  on  whether  the  sector  is  combined  with  another  sector,  or  on  the 
runway  configuration  being  used  at  BOS.  Each  runway  configuration  has  an  arrival  and 
departure  flow  that  is  specific  to  that  configuration.  Noise  abatement  and  environmental 
concerns  partially  determine  these  flows.  Supervisors  at  Boston  TRACON  routinely  combine 
sectors  when  workload  circumstances  warrant. 

For  the  current  study,  specialists  from  the  Technical  Center  developed  simulations  of  four  sectors 
based  on  actual  Boston  TRACON  sectors.  Descriptions  of  the  sectors  at  Boston  TRACON 
follow,  and  any  differences  between  the  actual  and  simulated  sectors  are  noted. 

a.  Initial  Departure.  All  aircraft  that  depart  BOS  use  the  Initial  Departure  Sector. 

Controllers  vector  aircraft  per  a  Logan-Nine  Standard  Instrument  Departure  procedure, 
which  outlines  departure  instructions  and  noise  abatement  procedures.  In  this  simulation, 
the  Initial  Departure  Sector  was  combined  with  the  Lincoln  Sector,  which  is  a  westbound 
departure  corridor  sector  and  an  inbound  sector  for  arrivals  from  the  southwest. 
(Controllers  hand  off  all  arrival  aircraft  from  the  southwest  to  the  Final  One  Sector  for 
sequencing  and  approach  clearances  to  BOS.) 

b.  South.  The  South  Sector  receives  departures  from  BOS,  including  both  jet  and  propeller 
traffic  departing  southbound.  In  this  simulation,  the  South  Sector  was  combined  with  the 
Plymouth  Sector,  which  is  predominantly  a  southbound  departure  corridor  and  an 
inbound  sector  for  arrival  flights  planned  over  Providence  or  from  the  Cape  Cod  area. 
(Controllers  vector  arrival  aircraft  to  runways  based  on  the  runway  configuration  in  use 
and  their  preference.  Controllers  hand  off  all  arrival  aircraft  to  the  Final  One  Sector  for 
sequencing  and  issuing  approach  clearances.) 
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c.  Rockport.  The  Rockport  Sector  is  mainly  a  north-  and  northeast-bound  departure  corridor 
and  an  inbound  sector  for  arrival  flights  planned  over  Gardner,  MA;  Manchester  and 
Pease,  NH;  or  the  Boston  overseas  arrival  fix,  25  nmi  east  of  the  airport.  The  Rockport 
Sector  receives  departures  from  the  Initial  Departure  Sector,  including  all  jet  and 
propeller  traffic  departing  to  the  north  and  northeast.  Controllers  vector  arrival  aircraft  to 
the  runway  in  use  and  then  hand  off  the  aircraft  to  the  Final  One  Sector  for  sequencing 
and  issuance  of  approach  clearances. 

d.  Final  One.  Final  One  is  the  final  approach  control  position  where  controllers  issue  all 
approach  clearances  for  BOS  and  subsequently  transfer  the  aircraft  to  the  Tower  Local 
Control  for  landing  clearances.  This  position  does  not  typically  control  departure  traffic, 
though  coordination  for  such  operations  may  be  requested.  Controllers  may  vector  an 
aircraft  to  any  runway  included  in  a  particular  configuration  for  a  more  efficient  use  of 
airspace  or  runway  utilization.  In  this  simulation,  the  Final  One  and  Final  Two  Sectors 
were  combined. 

Controller  participants  received  a  simulation  training  package  before  the  study.  This  package 
contained  detailed  information  on  the  airspace,  runway  configurations,  procedures,  and  controller 
actions  that  they  would  use  in  the  simulation.  The  briefing  package  also  included  the 
Background  Questionnaire  and  maps  of  the  airspace  and  runway  configurations.  Appendix  B 
contains  a  copy  of  this  package. 

2.3  Simulation  Scenarios 

Simulation  specialists  from  the  System  Simulation  and  Support  Branch  (ACT-510),  in 
collaboration  with  Boston  TRACON  personnel  and  engineering  research  psychologists  from 
ACT-530,  prepared  two  traffic  scenarios  that  were  representative  of  the  traffic  patterns  and 
characteristics  of  the  four  sectors.  These  scenarios  used  two  different  runway  configurations: 
Land  27/22L  -  Depart  22R  and  Land  4R/L  -  Depart  9.  These  scenarios  required  staffing  of  all 
four  sectors,  though  this  staffing  level  was  lighter  than  a  typical  90*'’  percentile  day  at  Boston 
TRACON.  There,  two  controllers  typically  staff  the  Final  One  sector,  and  one  controller  staffs  a 
satellite  position,  for  a  total  of  six  controllers.  Personnel  and  equipment  availability  limited  the 
staffing  that  could  be  used  in  the  simulation. 

The  traffic  volume  in  the  scenarios  was  equivalent  to  a  90'^'  percentile  day  at  Boston  TRACON 
with  density  varying  from  moderate  to  heavy.  Researchers  believed  that  this  traffic  volume 
would  be  sufficient  to  functionally  exercise  the  ARTS  IHA.  Simulation  specialists  at  the  • 
Technical  Center  developed  the  scenarios  from  Continuous  Data  Recording  (CDR)  tapes 
recorded  at  Boston  TRACON  on  July  25, 1995,  between  the  hours  of  1400  and  1600  local  time. 
Specialists  from  the  Boston  TRACON  training  department  verified  and  rated  the  scenarios  and 
tested  them  in  the  Technical  Center  laboratories.  Both  scenarios  contained  a  mix  of  jet  and 
propeller-driven  aircraft  flying  IFR  flight  plans  that  either  originated  or  terminated  service  at 
BOS.  Including  VFR  flight  plan  aircraft  or  overflight  aircraft  was  not  technically  feasible  given 
the  platform  and  timeframe  of  the  simulation  (see  Section  4. 1 .2). 

The  scenarios  originally  did  not  include  any  special  events  or  unscripted  pilot  requests  so  as  not 
to  reduce  the  repeatability  of  the  simulation.  The  researchers  believed  that  inclusion  of  these 
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events  could  have  focused  simulation  timing  and  controller  preferences  on  techniques  for 
handling  problems  rather  than  on  routine  ATC  operations.  However,  during  the  conduct  of  the 
simulation,  the  researchers  observed  that  controller  workload  was  not  as  high  as  expected  in  runs 
using  the  4R/L  runway  configuration.  The  research  psychologist  managing  the  activity,  in 
collaboration  with  supervisors  from  Boston  TRACON,  decided  to  increase  the  taskload  by 
closing  one  of  the  runways  in  this  configuration.  The  researchers  introduced  this  event  to 
increase  complexity,  add  variety  to  the  simulation,  and  provide  a  more  challenging  problem  in 
which  to  assess  controller  performance  and  workload.  The  researchers  believed  this  change 
would  require  additional  traffic  management  especially  by  arrival  sector  controllers.  This  special 
event  occurred  as  follows:  About  45  min  into  the  90-min  run,  the  expert  observer  announced  that 
bad  weather  and  poor  visibility  had  caused  the  closing  of  the  4L  runway.  This  weather  situation 
forced  all  traffic  to  land  at  the  4R  runway  for  the  remainder  of  the  run.  Section  3.2. 1  describes 
the  effect  of  this  manipulation  on  controller  workload. 

2.4  Laboratory  Platform 

The  Technical  Center  ARTS  EUA  Laboratory  served  as  the  primary  data  collection  site  for  this 
study.  Participants  controlled  traffic  using  four  ARTS  DIA  consoles.  The  expert  observer  could 
monitor  the  traffic  situation  from  a  fifth  console.  The  lighting  conditions  in  the  laboratory  were 
realistic  compared  to  the  levels  at  Boston  TRACON.  Specialists  from  the  laboratory  ensured  that 
all  radar  consoles  and  communication  equipment  functioned  properly. 

The  TGF  provided  simulated  airspace  and  targets.  Simulation  specialists  from  the  TGF  ensured 
that  the  scenarios  ran  smoothly  and  that  all  simulation  equipment  functioned  properly.  SIMOPs 
from  the  TGF  controlled  simulated  aircraft  using  special  workstations  and  made  simulated  air- 
ground  communications  with  controllers  using  the  Amecom  system.  SIMOPs  also  made 
simulated  ground-ground  communications  if  controllers  required  coordination  with  other 
facilities  or  sectors. 

2.5  Simulation  Schedule 

The  study  began  the  week  of  September  18,  1995  and  continued  for  3  consecutive  weeks.  Each 
week  involved  a  new  group  of  four  controllers  and  a  new  expert  observer.  On  the  first  day  of 
each  week,  controllers  and  expert  observers  received  a  pretest  briefing,  a  tour  of  the  ARTS  IHA 
Laboratory,  and  an  introduction  to  the  data  collection  techniques  and  equipment.  Controller 
participants  received  briefings  on  all  laboratory  and  data  collection  equipment  and  procedures. 
These  briefings  focused  on  issues  of  confidentiality  and  informed  consent,  particularly  as  these 
issues  relate  to  the  audio  and  video  recordings  made  during  the  simulation  runs.  On  the  second, 
third,  and  fourth  days  of  each  week,  controllers  completed  two  or  three  simulation  runs  per  day. 
On  the  fourth  day,  controllers  received  a  final  briefing.  Testing  ended  on  October  5,  1995. 

During  the  3  weeks  of  testing,  there  were  24  successful  simulation  runs  (7  runs  during  the  first,  9 
during  the  second,  and  8  during  the  third  week).  This  resulted  in  a  large  data  set  and  a  reliable 
baseline.  Laboratory  hardware  problems  forced  researchers  to  abort  some  runs,  which  resulted  in 
an  uneven  number  of  runs  from  week  to  week.  Each  run  lasted  90  min  and  alternated  the  two 
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runway  configurations.  Each  controller  staffed  a  different  sector  during  each  run  so  that  they 
staffed  every  combination  of  sector  and  runway  configuration  at  least  once  during  the  week. 

2.6  Objective  and  Subjective  Measures 

The  TGF  and  CDR  systems  recorded  objective  data.  These  measures  focused  on  quantifying 
traffic  volume,  flight  duration,  traffic  characteristics,  and  other  factors  in  each  sector.  Another 
goal  for  recording  objective  data  was  to  determine  the  input/output  activity  at  each  sector 
position  to  measure  how  each  controller  used  the  system. 

Five  questionnaires,  completed  by  controllers  and  expert  observers,  provided  subjective  data. 

The  Background  Questionnaire,  which  was  part  of  the  Briefing  Document  (Appendix  B),  focused 
on  the  experience  levels  and  other  pertinent  data  from  the  controller  participants.  The  Post- 
Scenario  Questionnaire  contained  queries  on  perceived  overall  workload,  problem  difficulty, 
self-ratings  of  performance,  and  simulation  realism.  The  Final  Questionnaire  addressed 
workstation  and  display  ergonomics  and  included  space  for  written  comments.  Expert  observers 
rated  controller  performance  using  the  Observer  Evaluation  Form  developed  by  Sollenberger  et 
al.  (1997)  and  kept  notes  on  simulation  technical  problems  using  the  Observer  LxDg.  Appendix  A 
provides  copies  of  these  four  questionnaires. 

The  researchers  used  four  Workload  Assessment  Keypads  (WAKs)  to  measure  subjective 
controller  workload  using  the  Air  Traffic  Workload  Input  Technique  (ATWIT)  (Stein,  1985). 
Each  WAK  consisted  of  a  box  with  small,  lighted  keys  (numbered  1  through  7)  and  a  tone 
generator.  Each  WAK  was  connected  to  one  of  four  laptop  computers  that  controlled  the  timing 
of  prompts  and  recorded  data.  The  WAKs  were  positioned  on  the  ARTS  IDA  console  and  could 
be  repositioned  according  to  controller  preference.  Every  5  min  during  a  simulation  run,  each 
WAK  emitted  a  short  beep  and  illuminated  its  lights,  prompting  controllers  to  rate  their  workload 
from  1  (low)  to  7  (high).  Entry  of  a  workload  rating  caused  the  WAK  lights  to  extinguish  until 
the  next  prompt.  The  laptop  computers  recorded  these  ratings  automatically.  If  a  controller  did 
not  enter  a  workload  rating,  the  lights  remained  illuminated  for  20  seconds  and  then 
extinguished.  In  such  cases,  the  laptop  computers  recorded  a  workload  rating  of  10  (a  missing 
data  code). 

Three  small  video  cameras  recorded  controller  activities.  Two  cameras,  positioned  above  and 
behind  the  controllers’  workstations,  recorded  their  physical  actions  (e.g.,  display  adjustments, 
trackball  and  keyboard  use,  and  WAK  entries)  but  could  not  record  information  displayed  on  the 
controllers’  screens.  A  third  camera  recorded  the  display  of  a  single  ARTS  TTTA  console  showing 
all  four  sectors.  Videotapes  recorded  the  voices  of  the  controllers.  Researchers  reviewed  the 
videotapes  as  part  of  the  data  analysis  to  validate  start  times,  controller  positions,  and  so  forth. 
Researchers  also  used  videotapes  to  review  loss  of  separation  incidents.  Appendix  C,  Table  C-1 
describes  this  analysis. 

The  data  sources  employed  for  this  testing  activity  were  the 

a.  Background  Questionnaire  (completed  at  the  beginning  of  the  week), 

b.  Post-Scenario  Questionnaire  (completed  after  each  simulation  run). 
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c.  Final  Questionnaire  (completed  at  the  end  of  the  week), 

d.  Observer  Evaluation  Form  (completed  once  per  controller  during  the  week), 

e.  Observer  Log  (completed  during  each  simulation  run), 

f.  Amecom  audio  tape  from  communication  system, 

g.  real-time  controller  workload  rating  (ATWIT), 

h.  videotape  with  audio, 

i.  TGF  data  recording,  and 

j.  CDR  tape. 

The  definitions  for  each  of  the  baseline  measures,  including  their  categorization  by  operational 
construct  and  the  rationale  for  use  in  baselining  the  ARTS  niA,  are  as  follows.  (The  source  for 
each  measure  is  usually  indicated  in  parentheses.) 

a.  Safety 

1 .  Operational  Errors  was  a  basic  safety  measure  representing  loss  of  applicable 
separation  minima.  (TGF) 

2.  Conflict  Alerts  was  a  system-initiated  display  warning  the  controller  of  imminent 
aircraft-to-aircraft  conflicts.  The  conflict  alert  system  had  features  to  minimize  the 
false  alarm  rate  in  the  terminal  area.  (CDR) 

3.  Other  Safety-Critical  Issues  were  derived  from  expert  observer  comments  on  system 
safety  issues  and  deficiencies.  (Observer  Log) 

b.  Capacity 

1 .  Aircraft  Under  Control  was  a  basic  capacity  measure.  It  represented  a  tally  of  traffic 
under  track  control.  (TGF) 

2.  Average  Time  in  Sector  (Handoff  to  Handoff)  was  a  measure  of  sector  efficiency. 
Increased  time  in  sector  may  have  indicated  less  efficient  movement  of  aircraft  in 
the  airspace  or  controller-induced  delay  vectoring  due  to  a  traffic  overload  situation. 
(TGF) 

3.  Average  Time  in  Sector  (Arrivals)  was  a  measure  of  arrival  sector  efficiency.  (TGF) 

4.  Average  Time  in  Sector  (Departures)  was  a  measure  of  departure  sector  efficiency. 
(TGF) 

5.  Aircraft  Spacing  on  Final  Approach  was  a  measure  of  the  efficiency  of  the  traffic 
flow  on  final  approach.  This  measure  represented  the  distance  from  an  aircraft  over 
the  middle  marker  to  the  aircraft  immediately  trailing  it.  Large  and  variable  spacing 
could  indicate  differences  in  control  style  and  changes  in  traffic  density.  (TGF) 

6.  Minutes  Between  Arrivals  was  a  measure  of  the  traffic  density  on  final  approach. 
This  measure  represented  the  minutes  that  elapsed  between  consecutive  aircraft 
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passing  over  the  middle  marker.  Shorter  times  between  landings  could  indicate 
increased  traffic  density.  (TGF) 

7.  Altitude  Assignments  Per  Aircraft  provided  a  ratio  of  total  altitude  assignments  to 
number  of  aircraft  under  control.  It  was  an  indicator  of  the  relative  efficiency  of 
aircraft  movement  through  the  sector.  Controllers  commonly  relied  on  vertical 
separation  in  preference  to  vectoring  solutions.  This  resulted  in  level-offs  and  climb 
or  descent  delays.  A  decrease  in  altitude  assignments,  with  a  corresponding 
decrease  in  climb  or  descent  delays,  could  indicate  greater  efficiency.  An  increase 
in  altitude  assignments  with  a  corresponding  increase  in  climb  or  descent  delays  and 
level-offs  could  indicate  less  efficiency.  (TGF) 

c.  Performance 

1 .  Data  Entries  was  a  relative  measure  of  data  entry  workload  for  the  controller 
position.  (CDR) 

2.  Data  Entry  Errors  was  a  relative  measure  of  data  entry  effectiveness.  Significant 
variations  may  indicate  difficult  message  syntax,  awkward  entry  device  layout,  or 
other  possible  factors.  (CDR) 

3.  Number  of  Altitude,  Speed,  and  Heading  Changes  represented  the  efficiency  of 
sector  operations  for  total  number  of  clearances  issued  in  these  three  categories. 
Significant  variation  in  relative  proportions  could  show  controllers  had  changed 
their  method  for  handling  traffic.  These  counts  were  based  upon  aircraft-related 
data  entries  at  the  SIMOP  positions.  (TGF) 

4.  ATC  Services  were  measures  of  the  quality  of  ATC  services  and  indicators  of 
system  performance.  Controllers  made  ratings  on  the  Post-Scenario  Questionnaire 
that  ranged  from  1  (low)  to  8  (high).  The  specific  items  composing  the  measure 
were  the  rated  quality  of  ATC  services  from  (a)  the  pilot’s  perspective  and  (b)  the 
controller’s  perspective. 

5-  Human  Capabilities  for  ATC  were  measures  representing  human  capabilities  used 
by  the  controller  in  performing  ATC  functions.  Expert  observers  made  ratings  on 
the  Observer  Evaluation  Form  that  served  as  indicators  of  operator  efficiency  and 
effectiveness  based  on  a  1  (low)  to  8  (high)  scale.  They  were  encouraged  to 
comment,  and  a  form  was  provided  for  that  purpose.  The  rating  scales  are  more 
completely  described  in  Sollenberger  et  al.  (1997).  The  specific  items  composing 
the  measure  assessed 

a)  how  well  the  controller  maintained  safe  and  efficient  flow, 

b)  how  well  the  controller  maintained  attention  and  vigilance, 

c)  how  well  the  controller  prioritized, 

d)  how  well  the  controller  communicated  and  informed,  and 

e)  the  level  of  the  controller’s  technical  knowledge. 
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d.  Workload 


1.  Workload  Per  Aircraft  was  a  measure  that  estimated  the  amount  of  workload 
expended  per  aircraft.  Subjective  workload  ratings  corresponded  closely  to  the 
number  of  aircraft  tracked  throughout  the  baseline  scenarios.  (ATWIT  and  TGF) 

2.  Average  Workload  was  the  mean  subjective  workload  reported  by  controllers,  by 
sector,  across  the  entire  simulation.  Workload  is  the  human  response  to  the 
demands  or  task  loads  produced  by  the  airspace  system.  Human  response  consisted 
of  observable  control  actions  and  cognitive  activity.  (ATWIT) 

3.  Post-Run  Workload  was  a  measure  of  average  workload  for  the  scenario  as  part  of 
the  Post-Scenario  Questionnaire.  The  rating  scale  ranged  from  1  (low)  to  8  (high). 

4.  Communication  Workload  was  the  mean  number  of  push-to-talk  communications 
per  aircraft  worked.  This  measure  detected  changes  in  communication  workload 
needed  to  control  aircraft.  Increased  communications  per  aircraft  may  have 
indicated  a  less  efficient  automation  interface.  Conversely,  increased 
communications  per  aircraft  may  have  represented  greater  latitude  for  controllers  to 
maneuver  aircraft  and  initiate  actions.  (TGF) 

5.  Data  Entry  Workload  was  the  mean  number  of  data  entries  per  aircraft  worked  and 
detected  changes  in  workload  required  to  control  aircraft.  (CDR  and  TGF) 

e.  Usability 

1 .  ARTS  TTTA  Console  were  measures  of  the  usability  of  the  system  as  rated  by 
controllers.  These  ratings  ranged  from  1  (low)  to  8  (high).  The  specific  items 
composing  these  measures  on  the  Final  Questionnaire  assessed 

a)  how  easily  the  controller  can  access  controls; 

b)  how  intuitively  controllers  operate  controls; 

c)  how  easily  controllers  use  the  keyboard; 

d)  how  easily  controllers  read  radar  and  map  displays; 

e)  how  easily  controllers  understand  radar  and  map  displays; 

f)  the  sufficiency  of  the  workstation  space; 

g)  how  well  equipment,  displays,  and  controls  support  efficient  ATC; 

h)  the  amount  of  limitation  imposed  by  equipment,  displays,  and  controls; 

i)  the  overall  effectiveness  of  equipment,  displays,  and  controls;  and 

j)  the  overall  quality  of  interaction  with  equipment. 

f.  Simulation  Fidelity 

1 .  Traffic  Characteristics  was  a  measure  representing  the  scenario  length,  number  of 
flights,  type  of  flight  (arrival,  departure,  or  overflight),  and  type  of  aircraft  (jet  or 
propeller).  It  was  a  characterization  of  the  simulation  scenario.  (TGF) 
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2.  Perceived  Representativeness  was  a  measure  of  the  controllers’  perceived  fidelity  of 
the  simulation  scenarios  for  the  four  sectors.  It  was  a  check  on  the  realism  of  the 
simulation.  These  ratings  ranged  from  1  (low)  to  8  (high).  The  items  comprising 
this  measure  on  the  Post-Scenario  Questionnaire  were 

a)  realism, 

b)  technical  problems,  and 

c)  problem  difficulty. 

3.  Summary  Data 


The  purpose  of  this  study  was  to  develop  a  baseline  of  performance  data  typifying  the  existing 
ARTS  IDA  system.  It  was  the  intention  that  these  data  be  used  for  comparisons  with  new 
systems  designed  for  the  terminal  environment.  If  the  conditions  of  this  study  were  duplicated,  it 
should  be  possible  to  compare  the  systems  using  the  measures  specified  in  this  report.  However, 
it  is  not  expected  that  the  measures  described  and  enumerated  here  represent  a  final  set. 
Operational  and  human  engineering  judgement  should  be  employed  in  their  application. 

By  itself,  this  report  has  limited  value,  except  as  an  exercise  in  baselining  an  operational  FAA 
system.  It  should  be  treated  as  a  database  of  information  that  forms  the  foundation  for  future 
baselining  efforts  and  system  comparisons.  When  future  systems  are  measured  using  the  same 
approach,  then  useful  comparisons  and  insights  will  be  gained  about  their  strengths  and 
weaknesses  using  a  basis  of  objective  and  subjective  measures. 

3. 1  Measure  Summary  Data 

Appendix  C,  Table  C-1  provides  a  summary  of  all  measures  aggregated  across  all  sectors, 
intervals,  and  corresponding  simulation  runs.  It  also  provides  short  descriptions  of  each  measure. 
For  some  measures,  the  table  presents  the  aggregated  data  and  refers  to  more  detailed 
information  contained  in  Tables  C-2  through  C-24.  This  additional  information  is  intended  to 
augment  the  aggregate  data  (e.g.,  to  assess  differences  between  sectors,  runway  configurations,  or 
time  intervals).  For  some  measures.  Table  C-1  indicates  that  aggregate  data  are  not  meaningful 
and  refers  to  other  tables  containing  the  pertinent  data. 

Appendix  D  lists  the  narrative  responses  made  by  the  controller  participants  on  the  Final 
Questionnaire.  These  items  address  issues  of  usability  and  performance  of  the  ARTS  IDA  and 
simulation  fidelity. 

3.2  Sector  Summary  Data 

Tables  C-2  and  C-4  provide  the  means  for  each  sector  and  runway  configuration,  aggregated 
across  15-min  intervals  and  simulation  runs.  Table  C-2  provides  the  means  for  quantitative  data 
and  C-4,  for  questionnaire  data.  Some  measures  contain  references  to  other  tables  providing 
additional  data.  Tables  C-3  and  C-5  provide  the  standard  deviations  at  the  sector  summary  level 
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for  the  quantitative  and  questionnaire  measures.  Table  C-6  provides  the  mean  number  of  ARTS 
entries  for  each  sector,  runway  configuration,  and  entry  type  aggregated  across  15-min  intervals 
and  simulation  runs. 

3.2.1  Workload  Manipulation 

Starting  in  the  second  week  of  the  study,  researchers  introduced  a  workload  manipulation  into  tlie 
runs  using  the  4R/L  runway  configuration.  Approximately  halfway  through  each  run,  poor 
visibility  forced  the  closing  of  the  4L  runway.  Researchers  examined  the  effect  of  this 
manipulation  on  the  workload  of  the  controller  staffing  the  Final  One  Sector  and  compared  the 
means  for  the  3  simulation  weeks  for  each  variable  making  up  the  Workload  construct  (workload 
per  aircraft,  average  workload,  post-run  workload,  communication  workload,  and  data  entry 
workload). 

An  analysis  of  variance  (ANOVA)  revealed  a  significant  main  effect  of  week  on  workload  per 
aircraft,  F(2,  21)  =  32.69,  p  <  .0001;  average  workload,  F(2,  21)  =  28.79,  p  <  .{X)01;  post-run 
workload,  F(2,  20)  =  6.86,  p  <  .01;  and  communication  workload,  F(2,  18)  =  9.16,  p  <  .01. 

There  was  no  significant  effect  on  data  entry  workload.  The  workload  manipulation  began  in  the 
second  week  so  an  effect  of  this  manipulation  would  appear  as  a  difference  between  the  first 
week  and  the  second  2  weeks.  The  Tukey-HSD  procedure  was  used  to  further  analyze  these 
main  effects  and  determine  which  weeks  differed. 

These  results  show  that  the  main  effects  of  week  are  unlikely  to  be  due  to  the  workload 
manipulation.  For  the  workload  per  aircraft,  average  workload,  and  post-run  workload  variables, 
the  main  effect  was  due  to  higher  workload  ratings  given  by  the  controllers  in  Week  3.  It  is 
unlikely  that  this  effect  is  due  to  the  workload  manipulation  because  controllers  in  Week  2  also 
experienced  the  manipulation  but  did  not  give  higher  workload  ratings  than  controllers  in 
Week  1.  For  the  communication  workload  variable,  the  main  effect  is  due  to  increased  numbers 
of  push-to-talk  communications  made  by  the  controllers  in  Week  2.  Again,  this  effect  is 
probably  not  due  to  the  workload  manipulation.  Controllers  in  Week  3  also  experienced  the 
manipulation  but  did  not  make  reliably  more  communications  than  controllers  in  Week  1.  The 
workload  manipulation  does  not  appear  to  have  affected  overall  workload  on  any  of  the  variables 
included  in  the  Workload  construct.  Because  of  this,  the  values  reported  in  Appendix  C  were 
collapsed  across  the  3  simulation  weeks. 

Examination  of  the  15-min  interval  workload  data  revealed  that  the  weather  manipulation 
primarily  changed  the  pattern  of  workload  ratings  rather  than  the  absolute  level.  Figure  1  shows 
the  average  workload  ratings  by  15-min  intervals  as  given  by  controllers  working  the  Final  One 
Sector  in  the  4R/L  runway  configuration.  In  Week  1,  workload  ratings  stayed  the  same  or 
decreased  between  45  to  60  min  into  the  simulation.  In  Weeks  2  and  3,  however,  workload 
ratings  stayed  the  same  or  increased  during  this  interval,  corresponding  to  the  onset  of  the 
weather  manipulation.  Tables  C-7  and  C-8  provide  a  subset  of  the  data  by  week  that  can  be  used 
to  further  examine  the  effect  of  this  manipulation. 
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Figure  1.  Workload  ratings  by  minutes  into  run  for  each  week. 

3.3  15-Minute  Interval  Summary  Data 

For  some  measures,  it  was  operationally  meaningful  to  separate  sector-level  statistics  by 
15-min  intervals.  For  example,  the  WAKs  collected  workload  data  every  5  min  during  the 
simulation  run.  Researchers  used  these  to  create  mean  ratings  for  each  15-min  interval.  Tables 
C-9  through  C-16  provide  these  means  for  each  15-minute  interval,  sector,  and  runway 
configuration  aggregated  across  simulation  runs.  Tables  C-17  through  C-24  provide  standard 
deviations  for  these  data. 

4.  Recommendations 


This  section  includes  information  on  the  application  of  these  data  for  system  comparisons.  It 
also  discusses  refinements  to  data  recording  and  analysis  procedures  that  future  baseline  studies 
should  use.  Finally,  it  addresses  limitations  or  constraints  that  apply  when  using  these  data. 

4.1  Use  of  Baseline  Data  for  System  Comparisons 

This  section  provides  guidance  on  using  this  baseline  measure  methodology  to  make 
comparisons  with  future  systems.  The  approach  taken  is  to  use  quantitative  baseline  measure 
data  in  combination  with  qualitative  information  to  assess  future  automation  systems. 

The  current  study  used  information  garnered  from  controllers  and  expert  observers  to  verify  any 
issues  or  concerns  identified  through  the  analysis  of  the  quantitative  data.  This  information  can 
also  identify  other  issues  or  concerns  not  captured  in  the  quantitative  measures.  It  may  also  be 
pertinent  in  the  comparison  of  a  future  system  to  the  current  baseline  system.  The  current  study 
obtained  this  information  during  simulation  run  debriefings  and  a  post-simulation  caucus. 
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The  current  study  represents  the  first  two  steps  of  a  five-step,  high-level  approach  as  follows: 

1 .  Collect  sufficient  data  on  the  current  system  to  provide  stable  estimates  on  all  specified 
operational  constructs  and  baseline  measures. 

2.  Reduce  and  analyze  the  data  collected  and  complete  the  tables  at  each  level  of  detail. 

3.  Collect  the  same  data  for  the  future  system  using  the  same  airspace,  simulation  scenarios, 
controllers  (if  possible),  and  other  aspects  of  the  simulation  that  might  otherwise  work  as 
intervening  or  confounding  variables. 

4.  Complete  the  identical  data  reduction  and  analysis  for  the  future  system. 

5.  Conduct  a  post-simulation  caucus  with  the  controllers  and  expert  observers  using  the  data 
comparisons  as  starting  points  to  identify  an  initial  set  of  issues  and  concerns.  Refer  to 
the  data  in  other  detailed  tables  to  augment  the  analysis  of  these  issues  and  data  contained 
in  observer  logs  and  debriefing  materials.  Make  systematic  comparisons  between  the 
terminal  baseline  and  the  future  system,  stepping  through  each  quantitative  measure. 
Examine  all  data  in  a  dynamic  fashion  to  identify  related  trends  that  may  or  may  not 
appear  in  other  operational  constructs  and  measures.  This  further  substantiates  or  refutes 
whether  a  problem  exists. 

During  the  caucus,  researchers  should  use  consensus-building  techniques  with  the  controllers  and 
observers  to  review  and  categorize  the  quantitative  comparisons,  identify  and  prioritize 
significant  issues,  and  assess  the  viability  of  potential  resolutions.  This  may  require  participation 
of  ATC  procedures  and  training  specialists.  As  part  of  the  assessment,  it  is  necessary  to  verify 
that  a  problem  is  not  an  artifact  of  the  simulation  platform  or  some  other  irrelevant  variable 
potentially  skewing  the  comparisons  between  the  two  systems. 

An  important  basis  for  determining  whether  the  future  system  is  comparable  to  the  baseline 
system  is  whether  the  data  for  any  particular  measure  are  statistically  equivalent.  That  is,  it  must 
be  determined  whether  the  two  systems  numerically  share  the  same  average  or  have  overlapping 
ranges  or  confidence  intervals.  However,  statistical  equivalence  or  nonequivalence  does  not 
automatically  indicate  operational  equivalence  or  nonequivalence.  Expert  judgment  must 
determine  this.  Results  can  fall  into  four  categories,  as  follows: 

a.  Category  1  involves  measures  where  the  baseline  and  future  systems  have  data  that  are 
statistically  equivalent  and  are  operationally  equivalent. 

b.  Category  2  involves  measures  where  the  baseline  and  future  systems  have  data  that  are 
statistically  equivalent  but  are  operationally  different. 

c.  Category  3  involves  measures  where  the  baseline  and  future  systems  have  data  that  are 
not  statistically  equivalent,  but  the  systems  are  operationally  equivalent. 

d.  Category  4  involves  measures  where  the  baseline  and  future  systems  have  data  that  are 
not  statistically  equivalent,  and  the  systems  are  operationally  different. 
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Traditional  descriptive  and  inferential  statistics  determine  statistical  equivalency.  A  preliminary 
approach  to  the  use  of  these  statistics  is  as  follows. 

a.  Compute  descriptive  statistics  making  general  comparisons  of  means,  standard 
deviations,  and  trends. 

b.  Derive  inferential  statistics  such  as  using  ANOVA  with  post  hoc  testing  to  compare  the 
baseline  and  future  systems  on  a  given  measure.  ANOVAs  will  be  two-way  tests 
comprised  of 

1 .  systems  (i.e.,  ARTS  lEA  baseline  versus  the  future  system),  and 

2.  a  second  factor  consisting  of  one  of  the  following: 

a)  four  sectors, 

b)  two  runway  configurations,  or 

c)  15-min  segments. 

The  ANOVA  first  checks  for  a  difference  in  each  of  the  factors  and  then  for  an  interaction 
between  the  two  factors.  If  the  ANOVA  reveals  statistically  significant  differences,  researchers 
should  use  post  hoc  testing  to  identify  where  the  difference(s)  occur. 

Researchers  should  adopt  an  alpha  level  (or  margin  for  error)  based  upon  an  operational 
projection  of  the  power  of  the  test.  They  should  assume  that  ATC  measures  are  normally 
distributed,  permitting  the  use  of  parametric  statistics.  Non-parametric  statistics  may  be 
appropriate  for  other  measures.  Statistical  tests  can  be  used  as  a  technique  to  compare  systems, 
but  they  do  not  eliminate  the  need  for  a  controller  caucus. 

An  example  demonstrating  the  use  of  an  ANOVA  is  to  consider  the  baseline  measure  of  the 
average  workload  for  the  terminal  controller.  Table  1  contains  the  means  for  this  measure  across 
the  four  sectors  in  the  27/22L  runway  configuration  and  shows  these  means  and  hypothetical 
means  for  a  future  system.  Figure  2  depicts  these  means. 


Table  1.  Average  Workload  Rating  by  System 


Initial  Dep. 

South 

Rockport 

Final  One 

ARTSmA 

3.0 

2.9 

3.7 

3.7 

Future  System  (hypothetical) 

4.5 

4.5 

3.0 

3.2 
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-  ARTS  IIIA 

•■Future  System 
(hypothetical) 


Sector 


Figure  2.  Average  workload  rating  by  sector  for  each  system. 

The  ANOVA  tests  for  an  overall  difference  between  the  ARTS  HLA  and  a  future  system  and  for 
differences  between  sectors.  It  also  tests  the  statistical  significance  of  the  interaction  represented 
in  Figure  2.  The  presence  of  an  interaction  means  that  there  is  a  differential  effect  in  how  a 
measure  such  as  workload  changes  across  the  two  variables  (systems  and  sectors).  If  the 
ANOVA  shows  significant  overall  effects  or  a  significant  interaction,  the  researcher  conducts 
post  hoc  tests  to  determine  where  the  difference(s)  occur.  This  might  show  that  the  hypothetical 
future  system  has  significantly  lower  workload  than  the  ARTS  IDA  for  the  arrival  sectors.  Even 
though  the  future  system  might  show  somewhat  higher  workload  values  for  the  departure  sectors, 
the  difference  may  not  reach  statistical  or  operational  significance.  Computational  techniques  for 
ANOVAs  are  readily  available  in  statistics  books  and  commercial  software  programs. 

4.1.1  Refinements  to  the  Baselining  Methodology 

For  future  efforts,  researchers  should  consider  the  following  enhancements  to  the  baseline  data 
extraction  and  analysis  process.  An  important  baseline  measure  for  capacity  is  aircraft  fuel 
consumption.  This  is  an  indicator  of  sector  efficiency  and  could  be  based  upon  sector  boundary 
crossing  time  in  contrast  to  track  control  time.  Fuel  consumption  could  be  measured  according 
to  average  pounds  of  fuel  consumed  for  all  aircraft,  by  sector.  Models  would  need  the  capability 
to  handle  TGF  or  CDR  output.  Researchers  should  also  collect  data  for  the  ARTS  IRA 
performance  time  of  functions  or  keyboard  entries  to  assist  in  the  evaluation  of  differences 
between  the  ARTS  IHA  CHI  and  that  of  a  future  system. 

During  the  preparation  of  this  report,  researchers  identified  the  need  for  additional  automated 
tools  to  expedite  data  reduction  and  analysis.  These  tools  would  be  used  offline  beginning  after 
completion  of  the  first  simulation  runs  and  in  parallel  during  the  remaining  simulation  runs.  In 
this  manner,  data  could  be  presented  in  a  timely  and  precise  manner  shortly  after  conclusion  of 
the  last  simulation  run.  A  particular  problem  was  the  extraction  and  analysis  of  CDR  output. 
Further  terminal  baseline  efforts  would  be  more  effective  if  improvements  were  made  in  the 
techniques  available  for  working  with  these  data. 
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4. 1 .2  Limitations  and  Constraints 


The  purpose  of  these  data  is  to  provide  a  baseline  for  future  system  comparisons  with  the  ARTS 
niA.  Neither  the  data  nor  the  constructs  upon  which  they  are  based  should  be  considered  as 
properly  validated  measures  for  use  in  other  studies  of  controller  or  system  performance.  Further 
research  is  needed  before  the  measures  described  in  this  report  could  be  used  for  applications 
other  than  terminal  system  baselining. 

CDR  was  not  fully  reliable  during  the  simulation  runs.  Of  the  24  successful  mns,  only  15 
contained  complete  CDR  output.  Chi-square  analyses  were  conducted  to  determine  whether 
proportions  of  data  in  this  smaller  set  were  biased  toward  a  particular  runway  configuration  or 
data  collection  week.  The  15  runs  of  CDR  output  are  as  representative  of  the  two  runway 
configurations  as  the  full  24  runs  of  questionnaire,  ATWIT,  and  TGF  data,  x^(l,  A^=  15)  =  0.21, 
p  >  .05.  Table  2  shows  the  percentage  of  each  runway  configuration  for  the  full  and  CDR  data 
sets.  The  15  runs  of  CDR  output  are  as  representative  of  the  3  weeks  as  the  full  data  set, 

X^(2,  A^=  15)  =  1.30,  p>  .05.  Table  3  shows  the  percentage  of  each  week  for  the  full  and  CDR 
data  sets. 


Table  2.  Representativeness  of  Data  Sets  for  Each  Runway  Configuration 


Percentage  of 
Full  Data  Set 

Percentage  of 
CDR  Data  Set 

27/22L  Runway 
Configuration 

46% 

40% 

4R/L  Runway 
Configuration 

54% 

60% 

Table  3.  Representativeness  of  Data  Sets  for  Each  Week 


Percentage  of 
Full  Data  Set 

Percentage  of 
CDR  Data  Set 

Week  1 

29% 

20% 

Week  2 

38% 

33% 

Week  3 

33  % 

47% 

As  a  result,  for  conflict  alert  and  keyboard  entry  data,  the  reported  means  and  standard  deviations 
are  based  on  15  completed  runs  rather  than  the  full  24.  This  smaller  sample  probably  resulted  in 
increased  variance  for  these  variables.  Efforts  should  be  made  in  future  ARTS  IRA  baseline 
work  to  ensure  more  reliable  performance. 

Though  this  simulation  attempted  thfe  highest  fidelity  available,  there  are  areas  in  which  it 
differed  from  the  actual  Boston  TRACON,  as  follows: 

a.  At  the  Boston  TRACON,  the  controller  working  the  Departure  Sector  has  a  closed-circuit 
television  display  showing  the  flight  strips  of  the  tower  controller.  The  controller  then 
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knows  the  call  signs  of  departure  aircraft  before  the  aircraft  arrives  in  the  terminal 
airspace.  This  simulation  did  not  have  this  capability. 

b.  Technical  limitations  and  limits  on  the  training  of  the  SIMOPs  prevented  the  inclusion  of 
VFR  traffic.  For  simulated  VFR  traffic  to  move  realistically  through  the  airspace, 
SIMOPs  would  have  needed  far  more  training  and  knowledge  of  the  terminal  area  and 
typical  VFR  flight  plans  than  were  available.  The  Boston  TRACON  would  typically 
handle  several  VFR  aircraft  during  a  90-min  period  on  a  90“’  percentile  day.  It  is  possible 
that  absolute  measures  (e.g.,  average  workload  and  total  data  entries)  are  lower  because 
fewer  aircraft  were  present  in  the  simulated  airspace  than  would  be  present  in  the  actual 
airspace.  However,  the  measures  reported  per  aircraft  should  be  mostly  unaffected  by  the 
lack  of  VFR  traffic. 

c.  The  staffing  used  in  the  simulation  (i.e.,  four  controllers  with  one  supervisor)  was  lighter 
than  a  typical  90'^  percentile  day  when  six  controllers  staff  the  positions.  However,  the 
exclusion  of  VFR  and  satellite  traffic  from  the  simulation  scenarios  made  the  staffing 
appropriate  for  the  traffic  load. 

These  limitations  do  not  affect  the  validity  of  the  data  set.  However,  when  making  comparisons 
with  future  systems,  researchers  should  maintain  similar  conditions. 

5.  Conclusions 


This  baseline  study  provides  a  data  set  that  should  be  useful  for  ensuring  that  new  ATC  systems 
function  as  well  or  better  than  the  existing  ARTS  IDA.  These  data  are  critical  as  a  foundation  for 
making  evaluations  that  would  otherwise  be  based  entirely  on  subjective  judgment.  If  used  as 
advised  in  this  report,  these  data  will  provide  a  powerful  tool  for  making  system  comparisons. 
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Appendix  A 

Questionnaires  and  Forms 


POST-SCENARIO  QUESTIONNAIRE 


Controller  ID:  a  b  c  d  Date _  Run 

Position/Sector:  South  Init.  Departure  Rockport  Final  Vector 

Test  System:  ARTS  IIIA  /  ARTS  HIE  /  STARS 


Instructions 

The  purpose  of  this  questionnaire  is  to  obtain  information  concerning  different  aspects  of  the  air  traffic 
control  problem  just  completed.  This  information  will  be  used  to  determine  how  the  simulation  experience  affects 
your  opinions.  As  you  answer  each  question,  feel  free  to  use  the  entire  numerical  scale.  Please  be  as  honest  and  as 
accurate  as  you  can.  So  that  your  identity  can  remain  anonymous,  your  actual  name  should  not  be  written  on  this 
form.  Instead,  your  data  will  be  identified  by  a  controller  code  known  only  to  yourself  and  the  experimenters. 


1)  How  well  did  you  control  traffic  during  this  problem? 

1  2  3  4  5 

Not  Very 
Well 


6  7  8 

Extremely 

Well 


2)  What  was  your  average  workload  level  during  this  problem? 

1  2  3  4  5  6 

Very  Low 
Workload 


7  8 

Very  High 
Workload 


3)  How  difficult  was  this  problem  compared  to  other  simulation  training  problems? 

1  2  3  4  5  6  7 

Not  Very 
Difficult 


8 

Extremely 

Difficult 


4) 


How  good  do  you  think  your  air  traffic  control  services  were  from  a  pilot's  point  of  view? 
1  2  3  4  5  6  7 

Not  Very 
Good 


8 

Extremely 

Good 


5) 


To  what  extent  did  technical  problems  with  the  simulation  equipment  interfere  with  your  ability  to  control 
traffic? 

1  2  3  4  5  6  7  8 


Not  Very 
Much 


A  Great 
Deal 


6)  To  what  extent  did  problems  with  simulator  pilots  interfere  with  your  normal  air  traffic  control  activities? 


1 

Not  Very 
Much 


8 

A  Great 
Deal 


7)  How  realistic  was  this  simulation  problem  compared  to  actual  air  traffic  control? 

1  2  3  4  5  6  7 

Not  Very 
Realistic 


8 

Extremely 

Realistic 


A4 


FINAL  QUESTIONNAIRE 


Controller  ID:  abed  Date 

Test  System:  ARTS  IIIA  /  ARTS  IIIE  /  STARS 


Section  A 

Please  circle  the  number  that  best  describes  your  level  of  agreement  with  each  of  the  following  statements 
concerning  the  current  ARTS  IIIA  console. 


1) 

The  switches,  knobs,  and  buttons  on  the  console  are  easy  to  access. 

1  2  3  4  5 

6 

7 

8 

Strongly 

Strongly 

Disagree 

Agree 

2) 

The  operation  and  functions  of  the  switches,  knobs,  and  buttons  on  the  console  are  intuitive. 

1  2  3  4  5 

6 

7 

8 

Strongly 

Strongly 

Disagree 

Agree 

3) 

The  controller  keyboard  is  easy  to  use. 

1  2  3  4  5 

6 

7 

8 

Strongly 

Strongly 

Disagree 

Agree 

4) 

The  radar  and  map  displays  are  easy  to  read. 

1  2  3  4  5 

6 

7 

8 

Strongly 

Strongly 

Disagree 

Agree 

5) 

The  radar  and  map  displays  are  easy  to  understand. 

1  2  3  4  5 

6 

7 

8 

Strongly 

Strongly 

Disagree 

Agree 

6) 

There  is  plenty  of  space  to  work  within  the  workstation. 

1  2  3  4  5 

6 

7 

8 

Strongly 

Strongly 

Disagree 

Agree 

7) 

The  equipment,  displays,  and  controls  allow  me  to  control  traffic  in  the  most  efficient 

way  possible. 

1  2  3  4  5 

6 

7 

8 

Strongly 

Strongly 

Disagree 

Agree 

8) 

The  equipment,  displays,  and  controls  allow  me  to  control  traffic  without  any  awkward  limitations. 

1  2  3  4  5 

6 

7 

8 

Strongly 

Strongly 

Disagree 

Agree 

9) 

Overall,  the  equipment,  displays,  anS  controls  are  effective  in  meeting  the  needs  of  controllers. 

1  2  3  4  5 

6 

7 

8 

Strongly 

Strongly 

Disagree 

Agree 

A>2 


FINAL  QUESTIONNAIRE 

(continued) 


Section  B 

Please  circle  the  number  that  best  describes  your  overall  interaction  with  the  equipment,  displays,  and 
controls  (i.e.,  human-computer  interface)  of  the  ARTS  IIIA  console.  In  making  these  judgments,  please  consider 
your  total  experience  with  the  ARTS  IILA,  not  just  your  experience  during  this  simulation  study. 

Regarding  my  everyday  air  traffic  control  tasks,  the  ARTS  IIIA  system  is: 


1) 

1  2 

Not  Very 

Limiting 

3 

4 

5 

6 

7  8 

Extremely 

Limiting 

2) 

1  2 

Not  Very 

Frustrating 

3 

4 

5 

6 

7  8 

Extremely 

Frustrating 

3) 

1  2 

Not  Very 

Effective 

3 

4 

5 

6 

7  8 

Extremely 

Effective 

4) 

1  2 

Not  Very 

Efficient 

3 

4 

5 

6 

7  8 

Extremely 

Efficient 

5) 

1  2 

Not  Very 

Easy  to  Operate 

3 

4 

5 

6 

7  8 

Extremely 

Easy  to  Operate 

6) 

1  2 

Not  Very 

Easy  to  Understand 

3 

4 

5 

6 

7  8 

Extremely 

Easy  to  Understand 
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FINAL  QUESTIONNAIRE 

(continued) 


Section  C 

Please  circle  the  number  that  best  represents  your  opinion  about  the  following  potential  improvements  to 
the  ARTS  IIIA. 


1) 


To  what  extent  do  you  think  a  "windows"  interface  similar  to  that  of  personal  computers  would  improve  your 
effectiveness  with  the  ARTS  IIIA  console? 


□  If  you  are  not  familiar  with  the  "windows"  interface,  mark  this  box. 

1  2  3  4  5  6 

7 

8 

Not  Very 

A  Great 

Much 

Deal 

2) 


To  what  extent  do  you  think  a  mouse  input  device  (instead  of  a  trackball)  would  improve  your  effectiveness  with 
the  ARTS  IIIA  console? 

□  If  you  are  not  familiar  with  a  mouse  input  device,  mark  this  box. 


1  2  3  4  5 

Not  Very 
Much 


6 


7 


8 

A  Great 
Deal 


3) 


To  what  extent  do  you  think  color  displays 
1  2  3 

Not  Very 
Much 


would  improve  your  effectiveness  with  the  ARTS  IIIA  console? 

4  5  6  7  8 

A  Great 
Deal 


4) 


To  what  extent  do  you  think  a  brighter  lighting  level  would  improve  your  effectiveness  with  the  ARTS  IIIA 
console? 


1  2  3  4  5 

Not  Very 
Much 


6 


7 


8 

A  Great 
Deal 
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FINAL  QUESTIONNAIRE 

(continued) 


Section  D 

For  each  the  following  questions,  indicate  your  opinion  by  marking  one  or  more  of  the  provided  boxes. 
Then,  please  provide  any  additional  comments  that  you  think  are  appropriate. 

1)  Which  aspects  of  the  ARTS  IIIA  console  need  improvement? 

□  Radar  and  Map  Displays  □  Console  Switches  and  Knobs 

□  Volume  of  Workspace  □  Trackball 

□  Keyboard  □  Other  (specify) _ 

□  Other  (specify) _ 

Please  provide  some  details  about  why  you  think  each  of  these  aspects  needs  improvement? 


2)  What  are  the  most  common  mistakes  you  encounter  using  the  ARTS  IIIA  console? 

□  Misreading  Radar  Display  Information  □  Selecting  Targets  with  Trackball 

□  Misreading  Map  Display  Information  □  Adjusting  the  Correct  Switch  or  Knob 

□  Making  Entries  with  Keyboard  □  Other  (specify) _ 

□  Other  (specify) _ 

Please  provide  some  details  about  what  you  think  causes  you  to  make  each  of  these  mistakes? 
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FINAL  QUESTIONNAIRE 

(continued) 


Section  E 

If  there  are  any  other  comments  or  suggestions  that  you  have  regarding  this  baseline  study  of  the  ARTS  IIIA 
console,  please  write  your  ideas  in  the  space  provided  below. 
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OBSERVER  EVALUATION  FORM 


Observer  Code _  Date 

Controller:  abed 

Position/Sector:  South  Init.  Depart  Rockport  Final  Vector 
Simulation _ 


INSTRUCTIONS 

This  form  was  designed  to  be  used  by  instructor  certified  air  traffic  control  specialists  to 
evaluate  the  effectiveness  of  controllers  working  in  simulation  environments.  Observers  will  rate 
the  effectiveness  of  controllers  in  several  different  performance  areas  using  the  scale  shown 
below.  When  making  your  ratings,  please  try  to  use  the  entire  scale  range  as  much  as  possible. 
You  are  encouraged  to  write  down  observations,  and  you  may  make  preliminary  ratings  during 
the  scenario.  However,  we  recommend  that  you  wait  until  the  scenario  is  finished  before  making 
your  final  ratings.  The  observations  you  make  do  not  need  to  be  restricted  to  the  performance 
areas  covered  in  this  form  and  may  include  other  areas  that  you  think  are  important.  Also,  please 
write  down  any  comments  that  may  improve  this  evaluation  form.  Your  identity  will  remain 
anonymous,  so  do  not  write  your  name  on  the  form.  Instead,  your  data  will  be  identified  by  an 
observer  code  known  only  to  yourself  and  the  researchers  conducting  this  study. 


Rating 

Label  Description 

1 

Controller  demonstrated  extremely  poor  judgment  in  making  control  decisions  and  very’  frequently  made 
errors 

2 

Controller  demonstrated  poor  judgment  in  making  some  control  decisions  and  occasionally  made  errors 

3 

Controller  made  questionable  control  decisions  using  poor  control  techniques  which  led  to  restricting  the 
normal  traffic  flow 

■ 

Controller  demonstrated  the  ability  to  keep  aircraft  separated  but  used  spacing  and  separation  criteria 
which  was  excessive 

5 

Controller  demonstrated  adequate  judgment  in  making  control  decisions 

6 

Controller  demonstrated  good  judgment  in  making  control  decisions  using  efficient  control  techniques 

■ 

Controller demonstrated  exce/Zenr  judgment  in  making  control  decisions  using  extremely  good 
control  techniques 

8 

Controller  always  demonstrated  excellent  judgment  in  making  even  the  most  difficult  control  decisions 
while  using  outstanding  control  techniques 

Not  Applicable  -  There  was  not  an  opportunity  to  observe  performance  in  this  particular  area  during  the 
simulation 
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MAINTAINING  SAFE  AND  EFFICIENT  TRAFFIC  FLOW 


1.  Maintaining  Separation  and  Resolving  Potential  Conflicts . 1 

-  using  control  instructions  that  maintain  safe  aircraft  separation 

-  detecting  and  resolving  impending  conflicts  early 

2.  Sequencing  Arrival  and  Departure  Aircraft  Efficiently . 1 

-  using  efficient  and  orderly  spacing  techniques  for  arrival  and  departure 
aircraft 

-  maintaining  safe  arrival  and  departure  intervals  that  minimize  delays 

3.  Using  Control  Instructions  Effectively . 1 

-  providing  accurate  navigational  assistance  to  pilots 

-  avoiding  clearances  that  result  in  the  need  for  additional  instructions  to 
handle  aircraft  completely 

-  avoiding  excessive  vectoring  or  over-controlling 

4.  Overall  Safe  and  Efficient  Traffic  Flow  Scale  Rating . 1 

MAINTAINING  ATTENTION  AND  SITUATION  AWARENESS 

5.  Maintaining  Awareness  of  Aircraft  Positions . 1 

-  avoiding  fixation  on  one  area  of  the  radar  scope  when  other  areas  need 
attention 

-  using  scanning  patterns  that  monitor  all  aircraft  on  the  radar  scope 

6.  Ensuring  Positive  Control . 1 

7.  Detecting  Pilot  Deviations  From  Control  Instructions . 1 

-  ensuring  that  pilots  follow  assigned  clearances  correctly 

-  correcting  pilot  deviations  in  a  timely  manner 

8.  Correcting  Own  Errors  in  a  Timely  Manner . 1 

9.  Overall  Attention  and  Situation  Awareness  Scale  Rating . 1 

PRIORITIZING 

10.  Taking  Actions  in  an  Appropriate  Order  of  Importance . 1 

-  resolving  situations  that  need  immediate  attention  before  handling  low 
priority  tasks 

-  issuing  control  instructions  in  a  prioritized,  structured,  and  timely 
manner 

11.  Preplanning  Control  Actions . 1 

-  scanning  adjacent  sectors  to  plan  for  inbound  traffic 

-  studying  pending  flight  strips  in  bay 

12.  Handling  Control  Tasks  for  Several  Aircraft . 1 

-  shifting  control  tasks  between  several  aircraft  when  necessary 

-  avoiding  delays  in  communications  while  thinking  or  planning  control 
actions 

13.  Marking  Flight  Strips  While  Performing  Other  Tasks . 1 

-  marking  flight  strips  accurately  while  talking  or  performing  other  tasks 

-  keeping  flight  strips  current 

14.  Overall  Prioritizing  Scale  Rating . 1 


PROVroiNG  CONTROL  INFORMATION 


15.  Providing  Essential  Air  Traffic  Control  Information . 1  2  3  4  5  6  7  8  NA 

-  providing  mandatory  services  and  advisories  to  pilots  in  a  timely 

manner 

-  exchanging  essential  information 

16.  Providing  Additional  Air  Traffic  Control  Information . 1  2  3  4  5  6  7  8  NA 

-  providing  additional  services  when  workload  is  not  a  factor 

-  exchanging  additional  information 

17.  Overall  Providing  Control  Information  Scale  Rating . 1  2  3  4  5  6  7  8  NA 

TECHNICAL  KNOWLEDGE 

18.  Showing  Knowledge  of  LOAs  and  SOPs . 1  2  3  4  5  6  7  8  NA 

-  controlling  traffic  as  depicted  in  current  LOAs  and  SOPs 

-  performing  handoff  procedures  correctly 

19.  Showing  Knowledge  of  Aircraft  Capabilities  and  Limitations . 1  2  3  4  5  6  7  8  NA 

-  avoiding  clearances  that  are  beyond  aircraft  performance  parameters 

-  recognizing  the  need  for  speed  restrictions  and  wake  turbulence 

separation 

20.  Overall  Technical  Knowledge  Scale  Rating . 1  2  3  4  5  6  7  8  NA 

COMMUNICATING 

21.  Using  Proper  Phraseology . 1  2  3  4  5  6  7  8  NA 

-  using  words  and  phrases  specified  in  ATP  71 10.65 

-  using  ATP  phraseology  that  is  appropriate  for  the  situation 

-  avoiding  the  use  of  excessive  verbiage 

22.  Communicating  Clearly  and  Efficiently . 1  2  3  4  5  6  7  8  NA 

-  speaking  at  the  proper  volume  and  rate  for  pilots  to  understand 

-  speaking  fluently  while  scanning  or  performing  other  tasks 

-  clearance  delivery  is  complete,  correct  and  timely 

-  providing  complete  information  in  each  clearance 

23.  Listening  to  Pilot  Readbacks  and  Requests . 1  2  3  4  5  6  7  8  NA 

-  correcting  pilot  readback  errors 

-  acknowledging  pilot  or  other  controller  requests  promptly 

-  processing  requests  correctly  in  a  timely  manner 

24.  Overall  Communicating  Scale  Rating . 1  2  3  4  5  6  7  8  NA 
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MAINTAINING  SAFE  AND  EFFICIENT  TRAFFIC  FLOW 

1.  Maintaining  Separation  and  Resolving  Potential  Conflicts 


2.  Sequencing  Arrival  and  Departure  Aircraft  Efficiently 


3.  Using  Control  Instructions  Effectively 


4.  Other  Actions  Observed  in  Safe  and  Efficient  Traffic  Flow 


MAINTAINING  ATTENTION  AND  SITUATION  AWARENESS 

5.  Maintaining  Awareness  of  Aircraft  Positions 


6.  Ensuring  Positive  Control 


7.  Detecting  Pilot  Deviations  From  Control  Instructions 


8.  Correcting  Own  Errors  in  a  Timely  Manner 


9.  Other  Actions  Observed  in  Attention  and  Situation  Awareness 
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PRIORITIZING 

10.  Taking  Actions  in  an  Appropriate  Order  of  Importance 


11.  Preplanning  Control  Actions 


12.  Handling  Control  Tasks  for  Several  Aircraft 


13.  Marking  Flight  Strips  While  Performing  Other  Tasks 


14.  Other  Actions  Observed  in  Prioritizing 


PROVIDING  CONTROL  INFORMATION 

15.  Providing  Essential  Air  Traffic  Control  Information 


16.  Providing  Additional  Air  Traffic  Control  Information 


17.  Other  Actions  Observed  in  Providing  Control  Information 
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TECHNICAL  KNOWLEDGE 
18.  Showing  Knowledge  of  LOAs  and  SOPs 


19.  Showing  Knowledge  of  Aircraft  Capabilities  and  Limitations 


20.  Other  Actions  Observed  in  Technical  Knowledge 


COMMUNICATING 
21.  Using  Proper  Phraseology 


22.  Communicating  Clearly  and  Efficiently 


23.  Listening  to  Pilot  Readbacks  and  Requests 


24.  Other  Actions  Observed  in  Communicating 
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ARTS  Baseline  Test 
Observer  Log 

Observer  Initials: _  Date: _  Run: _ 

Position/Sector:  South  Init.  Departure  Rockport  Final  Vector 

Instructions:  Please  note  the  occurrence  of  missed  approaches  and  missed  ILS  acquisitions  by  noting 
system  time,  the  nature  of  the  event,  and  the  aircraft  involved.  Please  also  note  any  technical  problems 
and  other  safety-critical  or  otherwise  important  events.  Use  back  of  page  for  explanations,  if  necessary. 


System  Time 

Event 

Aircraft 
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Appendix  B 
Briefing  Document 


B-1 


1.0  INTRODUCTION 


The  intent  of  this  training  package  is  to  provide  air  traffic  controllers  with  a  working  knowledge 
of  the  selected  Boston  TRACON  airspace  that  will  be  employed  during  the  Terminal  Baseline 
Evaluation  using  the  ARTS  IHA.  The  testing  of  the  ARTS  niA  will  not  be  an  evaluation  of 
controllers’  skills.  These  simulations  are  part  of  an  ongoing  effort  to  assess  operational 
suitability  issues  related  to  future  air  traffic  control  (ATC)  systems. 

These  simulations  have  been  designed  to  enable  the  controller  to  enter  as  many  inputs  into  the 
system  as  possible.  The  intent  is  to  provide  “real  world”  situations.  Included  in  this  package  are 
general  descriptions  of  the  Boston  TRACON  sectors/positions  as  well  as  procedures  specific  to 
each  position  that  will  be  used  in  these  simulations. 

2.0  SIMULATION  ENVIRONMENT 


The  Target  Generation  Facility  (TGF)  consists  of  four  areas:  Target  Generator,  Simulation 
Operation  Pilots  (SIMOPs),  Exercise  Control,  and  Development  and  Support.  The  TGF 
interfaces  with  National  Airspace  System  (NAS)  automation.  The  function  of  the  TGF  is  to 
create  a  realistic  ATC  environment.  Aircraft  targets  will  respond  to  your  instructions  without 
question.  Each  time  you  call  an  aircraft,  it  should  respond  realistically. 

The  basic  design  of  the  system  is  to  provide  the  user  with  a  system  that  allows  the  controller  to 
issue  air  traffic  instructions.  It  should  also  have  each  aircraft  perform  in  a  manner  similar  to  a 
real  environment. 

2.1  TARGET  GENERATION  FACILITY 


The  TGF  is  interfaced  with  the  ARTS  IHA  and  Host  systems  and  is  designed  to  generate  digital 
radar  messages  for  a  simulated  airspace  environment. 

2.2  Simulation  Pilots 

The  SIMOPs  control  the  aircraft  target  during  the  simulation. 

2.3  Exercise  Control 


The  Exercise  Control  manages  the  execution  of  the  exercise. 

2.4  Development  and  Support 

The  Development  and  Support  area  includes  the  workstations  that  are  used  by  the  scenario 
development  analyst  to  develop  scenarios,  validate  the  data  base,  and  preview  the  scenario. 
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2.5  DOs  and  DON’Ts  of  the  System 


Do  not  expect  the  system  to  respond  to  you  as  an  aircraft  that  has  a  pilot  sitting  at  the  controls. 
This  means  that  your  clearances  must  be  technically  correct  in  format. 

Do  not  expect  logical  answers  to  questions  that  are  outside  the  actual  realm  of  control  of  the 
aircraft  to  which  your  SIMOP  is  responding.  The  SIMOPs  have  no  visual  reference  to  the 
movement  of  any  aircraft  in  the  sector.  They  do  have  access  to  much  of  the  information  you  will 
need  in  the  normal  routine  of  controlling  the  aircraft  involved  (i.e.,  indicated  airspeed,  altitude 
information,  heading,  and  distance  for  certain  fixes  along  their  filed  route).  They  also  can  supply 
you  with  aircraft  type,  equipment,  beacon  code,  and  destination.  Before  the  first  simulation  run 
begins,  it  is  suggested  that  you  brief  your  SIMOPs  on  typical  instructions  and  clearances  you  will 
be  using. 

2.6  Support 

There  is  a  group  of  developers  that  work  very  hard  to  provide  you  with  the  best  possible  system. 
There  is  a  constant  stream  of  enhancements  that  they  continue  to  work  on  throughout  the  year. 
When  you  leave  this  facility,  we  hope  to  have  your  ideas  and  suggestions  to  further  improve  the 
system. 

Another  group  that  continues  to  train  each  day  to  improve  for  you  is  the  SIMOPs.  Your 
relationship  with  the  SIMOP  is  very  important  to  them  and  the  success  of  the  scenarios  we  test. 
There  are,  however,  a  few  things  of  which  you  should  be  aware.  Although  the  training  required 
of  a  SIMOP  represents  a  sincere  effort  to  provide  you  with  realism,  they  are  not  professionally 
trained  air  traffic  controllers  and  most  of  them  have  no  pilot  experience. 

2.7  Ghost  Positions 

All  the  airspace  included  within  any  facility’s  area  must  be  accounted  for  in  a  given  simulation 
that  is  conducted  here  in  the  Technical  Center  Lab.  This  means  that  all  relevant  sectors  must  be 
included. 

There  are  two  additional  sectors  that  must  be  used  in  the  simulation  and  staffed  by  controllers. 
They  are  designated  as  “ghost  positions.”  One  sector  is  used  to  start  the  target  (inbound  ghost) 
and  the  other  is  used  to  terminate  the  target  (outbound  ghost). 

En  route  flights  initially  entering  the  facility’s  airspace  are  “started”  (start  track)  in  the  simulation 
at  a  programmed  time.  Flights  that  originate  inbound  to  the  scenario  are  started  (departed)  from 
the  inbound  ghost  sector.  Flights  that  are  terminated  within  the  facility’s  airspace  are  terminated 
(drop  track)  in  the  outbound  ghost  sector’s  airspace. 

When  a  flight  is  assigned  one  of  the  following  termination  frequencies,  the  SIMOP  enters  the 
frequency  into  the  TGF  computer,  and  the  radar  track  then  terminates,  following  6  additional 
minutes  of  flight.  The  following  frequencies  can  be  used  to  terminate  an  aircraft  when  it  exits  a 
scenario  sector  into  no-scenario  airspace  (adjacent  facility/sector).  When  the  aircraft  is  issued  the 
appropriate  frequency  by  the  controller,  the  SIMOP  will  enter  the  frequency  into  the  TGF 
computer.  The  frequencies  are  as  follows: 
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Boston  Center- 

133.42 

128.75  134.7 

Providence  Approach- 

133.85 

135.4 

Bradley  Approach- 

123.95 

Cape  Approach- 

118.2 

Manchester  Approach- 

118.8 

134.75  124.9 

3.0  SIMULATION  SCENARIO  DESCRIPTIONS 


Boston  TRACON  will  execute  a  developed  scenario  utilizing  four  radar  positions.  The  level  of 
traffic/complexity  is  mixed,  and  mostly  moderate  to  heavy.  Scenario  duration  is  approximately 
1.5  hours.  The  following  diagram  identifies  the  positions  and  associated  frequencies: 


General  Information 


Radar  Displays-  The  following  RADAR  displays  are  used  in  these  scenarios: 

Display  #6=Initial  Departure  Position 

Display  #7=South  Position 

Display  #9=Rockport  Position 

Display  #10=Final  One  Position 


Initial  Departure 


a.  “D”  position,  frequency  133.0. 

b.  this  position  utilizes  the  position  symbol  “D”. 

c.  combined  with  Lincoln  Sector  (“L”). 

d.  all  Boston  Departures  initiates  at  this  position. 

e.  all  arrivals  from  “WOONS”  are  handed  off  via 
interfacility  to  this  position. 


South  Sector 


a.  “S”  position,  frequency  120.6. 

b.  combined  with  Plymouth  Sector  (“M”). 

c.  this  position  utilizes  position  symbol  “S”  . 

d.  this  position  accepts  handoffs  from  “D”  destined  to 
SID  departure  points  of  “FRILL,”  “BURDY,”  “SEY,” 
“ACK,”  “HYA,”  “PVC,”  “LUCOS,”  “MVY,” 

and  “DRUNK.” 
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e.  this  position  will  accept  handoffs  from  interfacility  arrival  points  of 
“PVC,”  “FREDO and  “PVD.” 


Rockport  Sector  a.  “R”  position,  frequency  118.25. 

b.  this  position  utilizes  the  position  symbol  “R.” 

c.  this  position  accepts  handoffs  from  “D”  destined  to  SID  departure 
points  of  “MHT”  and  “PSM.” 

d.  this  position  accepts  handoffs  from  interfacility  arrival  points  of  “GDM,” 
“KHRIS,”  “RAYMY,”  and  “SCUPP.” 


Final  One  a.  “F”  position,  frequency  126.5. 

b.  this  position  utilizes  a  position  symbol  “F.” 

c.  this  position  accepts  only  intrafacility  handoffs. 


3.1  Initial  Departure  (“D”) 

In  these  scenarios,  this  position  is  combined  with  the  Lincoln  sector.  Lincoln  sector  is 
predominantly  a  westbound  departure  corridor  and  an  inbound  sector  for  arrivals  flight  planned 
over  “WOONS.” 


Frequency  Information 

Primary  frequency  for  this  position  is  133.0 

Departure  Procedures 

Initial  Departure  is  the  outlet  for  all  aircraft  departing  the  Logan  International  Airport.  Aircraft 
are  vectored  per  a  RADAR  Standard  Instrument  Departure  (Logan-Nine  SID)  procedure,  which 
outlines  departure  instructions  and  noise  abatement  procedures  as  follows: 

ALL  let  aircraft 

Runway  22R  or  22L:  Fly  heading  140  degrees,  climb  and  maintain  5,000’. 

Runway  9:  Fly  runway  heading,  climb  and  maintain  5,000’. 

Runway  4R:  Fly  runway  heading  until  the  BOS  4  DME,  then  turn  right  heading  090  degrees, 
climb  and  maintain  5,000’. 

ALL  prop  aircraft 

Ry  assigned  heading,  climb  and  maintain  3,000’. 
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Arrival  Procedures 


Since  the  configuration  of  Initial  Departure  combines  the  functions  of  “Lincoln  Sector,”  the 
following  arrivals  require  service. 

Route  Altitude 

WOONS  BOS  7,000’ 

Controller  Actions 


1.  All  arrival  aircraft  are  handed  off  to  the  Final  Vector  (“F”)  position  for  sequencing  and 
approach  clearances. 

3.2  South  Plymouth  (“S”) 

This  position  is  combined  with  the  Plymouth  Sector.  Plymouth  Sector  is  predominantly  a 
southbound  departure  corridor  and  an  inbound  sector  for  arrivals  flight  planned  over  “PVD,” 
“FREDO,”  and  “PVC.” 

Frequency  Information 

Primary  frequency  for  this  position  is  120.6 

Departure  Procedures 

Departures  are  handed  off  from  Initial  Departure  to  this  sector  for  jet/prop  traffic  departing 
southbound. 

To  Boston  Center 

Jet  departures  are  vectored  outbound  on  a  heading  of  170-210  degrees. 

Jet  departures  routed  over  ACK  (Nantucket)  are  issued  “direct  ACK.” 

Props  requesting  at  or  above  12,000’  are  issued  “maintain  12,000’”  and  vectored  on  a  heading  of 
170-210  degrees. 

To  Providence  Approach 

Props  requesting  at  or  below  10,000’  shall  be  vectored  to  join  V268  North  of  INNDY. 

To  Cape  Approach 

Props  landing  HYA,  MVY,  ACK  are  sent  via  “direct”  at  5,000’,  7000’,  or  9,000’. 

Props  landing  PVC  are  sent  via  “direct”  at  3,000’. 
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_^ival  Procedures 

The  following  arrivals  will  require  service  by  this  sector/position  landing  Boston. 


Runway 

Route 

4R 

PVD.V141.INNDY.BOS 

27 

PVD.V141.INNDY.BOS 

27 

FREDO.BOS 

27 

PVC.BOS 

AlHtude/Restriction(s) 
cross  PVD  at  1 1,000’,  at  250  knots 
cross  PVD  at  1 1,000’,  no  speed  restriction 
6,000’ 

4,000’ 


Controller  Actions 

1 .  Aircraft  may  be  vectored  to  either  4R  or  4L,  as  determined  by  controller  personnel  for  a  more 
efficient  use  of  airspace/runway  utilization. 

2.  All  arrival  aircraft  are  handed  off  to  the  Final  Vector  (“F”)  position  for  sequencing  and 
approach  clearances. 

3.3  Rockport  Sector  (“R”) 

Rockport  Sector  is  predominantly  a  north/northeast  bound  departure  corridor  and  an  inbound 
sector  for  arrivals  flight  planned  over  “GDM,”  “RAYMY,”  “KHRIS,”  and  “SCUPP.” 

Frequency  Information 

primary  frequency  for  this  position  is  1 18.25 

peparture  Procedures 

Departures  are  handed  off  from  Initial  Departure  to  this  sector  for  jet/prop  traffic  departing 
north/northeast  bound. 

To  Boston  Center 

Jet  departures  are  vectored  outbound  “direct  MHT”  or  “direct  PSM,”  as  appropriate, 
prop  departures  are  vectored  outbound  “direct  MHT”  or  “direct  PSM,”  as  appropriate, 
props  requesting  at  or  above  12,000’  are  issued  “maintain  12,000’”  and  vectored  “direct  MHT” 
or  “direct  “PSM,”  as  appropriate. 

To  Manchester  Approach 

prop  departures  to  Boston  Center  (at  or  above  12,000’)  may  be  issued  “maintain  10,000’”  and 
handed  off  to  Manchester  Approach.  Five  (5)  mile  longitudinal  separation  shall  be  provided  to 
these  successive  operations. 
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All  other  aircraft  will  be  issued  “direct  MHT”  and  climbed  to  10,000’  or  lower,  as  requested. 
Arrival  Procedures 


The  following  arrivals  require  service  by  this  sector/position  landing  Boston. 


Runway 

Route 

all 

RAYMY.LWM.BOS 

all 

KHRIS.LWM.BOS 

all 

GDM.V431.REVER.BOS 

all 

GDM.V431.REVER.BOS 

27 

SCUPP.BOS 

4R 

SCUPP.BOS 

all 

SCUPP.BOS 

Altitude/Restriction  (s) 

6,000’  (props) 

5,000’  (props) 

cross  BRONC  (props)  9,000’ 

cross  BRONC  (jets)  1 1,000’,  at  250  knots 

jets  1 1,000’,  at  230  knots 

jets  1 1 ,000’,  at  250  knots 

props  10,000’ 


Controller  Actions 


1.  Aircraft  may  be  vectored  to  either  4R  or  4L,  as  determined  by  controller  personnel  for  a  more 
efficient  use  of  airspace/runway  utilization. 

2.  Aircraft  may  be  vectored  to  either  22L  or  27,  as  determined  by  controller  personnel  for  a  more 
efficient  use  of  airspace/runway  utilization.  In  either  case,  ensure  aircraft  assigned  runway  22L 
by  controllers  are  capable  of  the  hold  short  operation  (simultaneously  landing  runway  27). 

3.  All  arrival  aircraft  are  handed  off  to  the  Final  Vector  (“F”)  position  for  sequencing  and 
approach  clearances. 

3.4  Final  One 

In  these  scenarios,  this  position  is  combined  with  the  Final  Two-(I)  position.  Final  One  is  the 
final  approach  control  position  where  all  approach  clearances  are  issued  for  Logan  International 
Airport  and  aircraft  are  subsequently  transferred  to  the  Tower  Local  Control  for  landing 
clearances.  This  position  does  not  typically  control  departure  traffic,  though  coordination  for 
such  operations  may  be  requested. 

Frequency  Information 

Primary  frequency  for  this  position  is  126.5 

Arrival  Procedures 


The  following  arrivals  require  service  by  this  sector/position  landing  Boston. 
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Runway 

Aircraft  Type  Transferred  By 

Altitude/Route 

4R 

jets 

Rockport  Sector 

6,000’/on  a  right  downwind 

4R 

jets 

South/Plymouth  Sector 

6,000’/established  on  the 
extended  use  4R  localizer 

4L 

props 

Rockport  Sector 

5,000’/on  a  left  downwind 

4L 

props 

Init.  Departure  from  WOONS 

4,000’/direct  BOS  VOR 

27/22L 

jets/props 

Rockport  Sector  from  GDM 

6,000’ /on  a  right  downwind 

27 

jets 

Rockport  Sector  from  SCUPP 

6,000’/vector  to  join  the  runway 
27  localizer 

27 

props 

Rockport  Sector  from  SCUPP 

5,000’/vector  to  join  the  runway 
27  localizer 

27 

jets 

South/Plymouth 

5,000’/left  base  leg  vector  at 
TONNI 

27 

props 

South/Plymouth 

4,000 ’/left  base  leg  vector  at 
TONNI 

22L 

props 

Rockport  Sector  from  RAYMY 

4,000’/right  base  leg  from  LWM 
VOR 

22L 

props 

Rockport  Sector  from  KHRIS 

4,000 ’/right  base  leg  from  LWM 
VOR 

22L 

props 

Init.  Departure  from  WOONS 

5,000’/right  downwind 

Controller  Actions 


1.  Aircraft  may  be  vectored  to  either  4R  or  4L,  as  determined  by  controller  personnel  for  a  more 
efficient  airspace/runway  utilization.  Aircraft  inbound  for  Runway  4L  should  be  vectored  for  the 
visual  approach  to  an  imaginary  final.  SIMOP  personnel  will  make  all  descents  and  necessary 
turns  after  the  issuance  of  the  visual  approach.  Runway  4R,  22L,  and  27  arrivals  shall  be 
vectored  for  that  runway’s  published  ILS  approach. 

2.  Primary  runway  arrivals  (runway  4R  or  27,  depending  on  configuration)  shall  remain  on  the 
position  symbol  “F.”  Secondary  arrivals  (runway  4L  or  22L)  data  tags  shall  be  changed  (local 
ARTS  patch)  to  a  position  symbol  of  “X.”  This  identifies  the  runway  assignment  and  reduces 
confusion  by  Approach/Tower  personnel. 
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3.5  Airspace  Descriptions 


A  general  outline  of  the  overall  airspace  delegated  to  Boston  TRACON  incorporating  internal 
sectorization  for  operations  for  the  Land  Runway  27/22L,  Depart  Runway  22R  configuration. 
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A  look  at  the  overall  flow  of  traffic  for  this  configuration, 
color  key 

red=JET  arrival  flow  (thick  line) 
blue=JET  departure  flow  (thick  line) 
orange=PROP  arrival  flow  (thin  line) 
green=PROP  departure  flow  (thin  line) 
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50 


60 


Land  4R/L,  Depart  9 


A  look  at  the  overall  flow  of  traffic  for  this  configuration. 

color  key 

red=JET  arrival  flow  (thick  line) 
blue=JET  departure  flow  (thick  line) 
orange=PROP  arrival  flow  (thin  line) 
green=PROP  departure  flow  (thin  line) 
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3.6  Additional  Scenario  Information 


Trust 


A  list  from  the  flight  plan  database  is  generated  using  a  script  developed  by  SRC  personnel.  See 
Stan  Rimdzius  or  Nizam  Taleb. 

Arrival  Handoff  Positions/Times 


Handoffs  of  arrival  aircraft  will  begin  approximately  30  seconds  after  target  initiation,  regardless 
of  inbound  Boston  airspace  sector  involved. 

Frequencies  Used  for  Interfacility  Handoffs  and  Termination 

Boston  Center  (implied  handoff  by  selecting  the  character  “C”  and  slewing): 

133.42-Bosox  Sector-(Bosox,  Clyde,  Nelie  at  or  above  1 1,000’) 

128.75- Cape  Sector-(SEY,  Lucos,  ACK,  HYA  at  or  above  1 1,000’) 

1 34.7- Concord  Sector-(MHT  at  or  above  1 1 ,000) 

128.2- Parso  Sector-(PSM  at  or  above  1 1,000’,  and  all  FRILL) 

Providence  Approach  (implied  handoff  by  selecting  “delta  1’’  and  slewing): 

133.85-Providence  East  High/Low-(BURDY,  V268,  east  satellites) 

135.4-Providence  West  High/Low-(all  west  satellites) 

Bradley  Approach  (implied  handoff  by  selecting  “delta  2”  and  slewing): 

123.95-Bradley  (Bosox,  Clyde  at  or  below  10,000’) 

Cape  Approach  (implied  handoff  by  selecting  “delta  3’’  and  slewing): 

1 18.2- Cape  HiglVLow  (HYA,  MVY,  ACK,  PVC  at  or  below  10,000’) 

Manchester  Approach  (implied  handoff  by  selecting  “delta  4”  and  slewing): 

1 18.8- Manchester  East  (PSM  at  or  below  10,000) 

134.75- Manchester  West  (MHT  5,000’  to  10,000’) 

124.9- Manchester  South  (landing  MHT,  ASH  at  below  4,000) 

Target  Termination 

Target  termination  occurs  6  minutes  after  interfacility  transfer  of  communications  has  occurred. 
This  ensures  that  the  aircraft  departs  Boston’s  airspace.  Use  of  any  of  the  above  interfacility 
frequencies  would  indicate  those  aircraft  requiring  this  action. 
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Voice  Communication  Equipment  Layout 


The  following  tables  identify  the  position  labeling  for  the  voice  communication  equipment.  The 
order  of  these  labels  should  be  consistent  with  the  following  tables  to  ensure  controller 
familiarity.  Foot  switches  should  be  incorporated  at  each  operating  position  for  optional  use  by 
controller  personnel. 


At  Radar  Display  #6  -Initial  Departure 

2050 

MHT2 

MHTT 

N 

S 

2151 

BDL 

OQU 

R 

L 

2154 

FMH 

FVl 

M 

HELO 

LS 

2307 

AM 

FV2 

LCW 

LCE 

RLS 

At  Radar  Display  #7  -South  Position 

5027 

MHT2 

MHTT 

N 

N*M 

NZW 

5028 

BDL 

OQU 

R 

L 

OWD 

2050 

FMH 

FVl 

M 

HELO 

LS 

2151 

D 

FV2 

LCW 

LCE 

RLS 

At  Radar  Display  #9  -Rockport  Sector 

5028 

MHT2 

MHTT 

N 

S 

2150 

BDL 

OQU 

AM 

L 

2151 

FMH 

FVl 

M 

HELO 

LS 

2154 

D 

FV2 

LCW 

LCE 

RLS 

At  Radar  Display  #] 

0  -Final  Vector 

D 

N 

S 

R 

L 

M 

HELO 

LS 

LCW 

LCE 

RLS 

Functionality 


Buttons  labeled  D,  S,  R,  and  FVl  should  have  voice  routed  to  those  respective  RADAR  positions 
identified.  These  should  serve  as  an  override  to  radio  frequency  transmissions. 

Buttons  labeled  L,  M,  FV2  serve  a  visual  function  in  labeling  only  and  are  not  routed  to  other 
positions/SIMOPs  at  this  time. 

All  remaining  buttons  are  routed  to  equipment,  scheduled  to  be  temporarily  placed  within  the 
SIMOP  area,  for  service  as  coordinated  with  ACT-510.  These  buttons  MAY  require  controller 
personnel  to  use  the  push-to-talk  feature  of  their  headsets  to  enable  conversation  with  that 
position/facility  called  up. 
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4.0  SIMULATION  SCHEDULE 


4.1  Week  1  Simulation  Schedule  and  Work  Assignments 

4.1.1  Definitions 


Simulation  A  Configuration:  Land  27/22L,  depart  22R 

Simulation  B  Configuration:  Land  4R/L,  depart  9 

Position  1  is  Initial  Departure 

Position  2  is  South  (combined  with  Plymouth  Sector) 

Position  3  is  Rockport  Sector 

Position  4  is  Final  Vector 

4.1.2  Day  1 


Pre-Briefing:  1400  - 1600 
Laboratory  Orientation:  1600  - 1800 

4.1.3  Day  2 

Meet  in  ARTS  Lab:  1530 

Simulation  Al:  1630  - 1830 
Position  Controller 

1  A 

2  B 

3  C 

4  D 


Break:  1830  - 1930 


Simulation  Bl:  1930  -  2100 
Position  Controller 

1  D 

2  C 

3  B 

4  A 


Simulation  A2:  2130  -  2300 
Position  Controller 


1 

2 

3 

4 


B 

C 

D 

A 


» 
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Meet  in  ARTS  Lab:  1530 


Simulation  B2:  1630  - 1830 
Position  Controller 


1 

2 

3 

4 


A 

D 

B 

C 


Break:  1830  - 1930 


Simulation  A3: 
Position 
1 
2 


1930  -  2100 

Controller 

C 

D 

A 

B 


Simulation  B3:  2130  -  2300 
Position  Controller 


1 

2 

3 

4 


B 

A 

D 

C 


4.1.5  Day  4 


Meet  in  ARTS  Lab:  1530 


Simulation  A4: 1630  - 1830 
Position  Controller 


1 

2 

3 

4 


D 

A 

B 

C 


Break:  1830  ■  1930 


Simulation  B4: 
Position 
1 
2 

3 

4 


1930  -  2100 

Controller 

C 

B 

A 

D 


Debriefing:  2130  -  2300 

4.2  Week  2  Simulation  Schedule  and  Work  Assignments 
4.2.1  Definitions 


Simulation  A  Configuration:  Land  27/22L,  Depart  22R 

Simulation  B  Configuration:  Land  4R/L,  Depart  9 

Position  1  is  Initial  Departure 

Position  2  is  South  (combined  with  Plymouth  Sector) 

Position  3  is  Rockport  Sector 

Position  4  is  Final  Vector 

4.2.2  Day  1 

Pre-Briefing:  1400  - 1600 
Laboratory  Orientation:  1600  - 1800 

4.2.3  Day  2 

Meet  in  ARTS  Lab:  1530 

Simulation  Bl:  1630  - 1830 
Position  Controller 

1  A 

2  B 

3  C 

4  D 

Break:  1830  - 1930 

Simulation  Al:  1930  -  2100 
Position  Controller 

1  D 

2  C 

3  B 

4  A 
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Simulation  B2:  2130  -  2300 
Position  Controller 


1 

2 

3 

4 


B 

C 

D 

A 


4.2.4  Day  3 


Meet  in  ARTS  Lab:  1530 


Simulation  A2:  1630  - 1830 


Position 

1 

2 

3 

4 


Controller 

A 

D 

B 

C 


Break:  1830  - 1930 

Simulation  B3: 1930  -  2100 
Position  Controller 

1  C 

2  D 

3  A 

4  B 


Simulation  A3:  2130  -  2300 
Position  Controller 

1  B 

2  A 

3  D 

4  C 

4.2.5  Day  4 

Meet  in  ARTS  Lab:  1530 

Simulation  B4: 1630  - 1830 
Position  Controller 


1 

2 

3 

4 


D 

A 

B 

C 
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Break:  1830  - 1930 


Simulation  A4: 
Position 
1 
2 

3 

4 


1930  -  2100 

Controller 

C 

B 

A 

D 


DebrieHng:  2130  -  2300 

4.3  Week  3  Simulation  Schedule  and  Work  Assignments 
4.3.1  Definitions 


Simulation  A  Configuration:  Land  27/22L,  Depart  22R 

Simulation  B  Configuration:  Land  4R/L,  Depart  9 

Position  1  is  Initial  Departure 

Position  2  is  South  (combined  with  Plymouth  Sector) 

Position  3  is  Rockport  Sector 

Position  4  is  Final  Vector 

4.3.2  Day  1 

Pre-Briefing:  1400  - 1600 
Laboratory  Orientation:  1600  - 1800 

4.3.3  Day  2 

Meet  in  ARTS  Lab:  1530 

Simulation  Al:  1630  - 1830 
Position  Controller 

1  A 

2  B 

3  C 

4  D 

Break:  1830  - 1930 
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Simulation  Bl:  1930  -  2100 
Position  Controller 


1 

2 

3 

4 


D 

C 

B 

A 


Simulation  A2:  2130  •  2300 
Position  Controller 


1 

2 

3 

4 


B 

C 

D 

A 


4.3.4  Day  3 


Meet  in  ARTS  Lab:  1530 


Simulation  B2:  1630  - 1830 
Position  Controller 


Break:  1830  - 1930 


Simulation  A3: 
Position 
1 
2 

3 

4 


1930  -  2100 

Controller 

C 

D 

A 

B 


Simulation  B3:  2130  >  2300 
Position  Controller 


1 

2 

3 

4 


B 

A 

D 

C 
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4.3.5  Day  4 


Meet  in  ARTS  Lab:  1530 


Simulation  A4: 1630  •  1830 
Position  Controller 


1 

2 

3 

4 


D 

A 

B 

C 


Break:  1830  - 1930 


Simulation  B4: 1930  -  2100 
Position  Controller 


1 

2 

3 

4 


C 

B 

A 

D 


Debriefing:  2130  -  2300 

4.4  Data  Collection 


Several  types  of  data  will  be  collected  to  provide  baseline  information  on  the  ARTS  TTTA 

4.4. 1  Computer-Recorded  Data 

Measures  of  such  items  as  number  of  aircraft  handled,  frequency  of  conflict  alerts,  and  number  of 
communications  will  be  collected  by  the  TGF,  ARTS  niA,  and  Amecom  (voice  switching) 
systems.  This  data  collection  is  automatic  and  requires  no  effort  from  the  controllers. 

4.4.2  Questionnaire  Data 

Your  opinions  on  the  usability  of  the  ARTS  IHA  will  be  requested,  along  with  other  types  of 
information,  using  questionnaires.  Please  complete  the  Background  Information  Questionnaire 
found  in  section  5  of  this  briefing  package.  Other  questionnaires  will  be  distributed  at  the  end  of 
each  run  and  at  the  end  of  the  third  day  of  simulation. 

4.4.3  Expert  Observer  Data 

Air  traffic  controllers  from  Boston  TRACON  and  other  locations  will  be  observing  the 
simulation  runs  and  recording  information  on  several  topics.  Among  other  things,  they  will  be 
evaluating  your  performance  on  a  number  of  scales  developed  by  the  Human  Factors  Laboratory. 
This  is  to  assess  how  well  the  ARTS  IDA  supports  you  in  your  work  and  as  a  basic  check  on 
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quality  of  performance.  This  information  will  remain  confidential  and  will  not  be  included  in 
any  report  materials. 

4.4.4  Workload  Data 

To  determine  the  baseline  characteristics  of  the  ARTS  IDA,  it  will  be  very  important  to  collect 
workload  data.  This  will  be  accomplished  using  a  workload  estimating  method  called  the  Air 
Traffic  Workload  Input  Technique.  A  keypad  will  be  positioned  at  your  workstation.  Every  4 
minutes,  you  will  be  prompted  by  auditory  and  visual  signals  to  enter  a  number  between  1  and  7 
on  the  keypad.  One  will  indicate  lowest  workload  and  7  will  indicate  highest  workload. 

5.0  CONSENT  FORM 


5.1  Purpose 

The  FAA  is  currently  in  the  process  of  procuring  new  terminal  air  traffic  control  systems  (i.e., 
STARS).  To  evaluate  the  relative  merits  of  these  new  systems,  we  are  collecting  baseline  data 
for  the  current  ARTS  IDA  console.  Later,  similar  data  will  be  collected  in  studies  of  the  future 
system.  As  you  work  the  air  traffic  control  problems  in  this  simulation,  data  will  be  recorded 
regarding  your  workload,  system  capacity,  and  system  performance.  The  purpose  of  these 
measures  is  not  to  evaluate  individual  controllers  but  to  determine  the  effectiveness  of  the  ARTS 
IDA  console.  Also,  you  will  be  asked  to  complete  several  questionnaires  requesting  your 
opinions  concerning  the  human-computer  interface  (i.e.,  workstation  equipment,  computer 
displays,  and  console  switches  and  knobs)  of  the  ARTS  IDA  console. 

5.2  Rights  of  Participants 

Please  understand  that  your  participation  in  this  study  is  strictly  voluntary,  and  your  right  to 
privacy  will  be  protected.  Your  responses  will  be  identified  by  a  participant  code  known  only  to 
yourself  and  the  experimenters.  No  individual  names  or  identities  will  be  recorded  or  released  in 
any  reports.  If  you  have  any  questions  at  any  time  regarding  the  study,  the  experimenters  will  be 
happy  to  answer  them. 

5.3  Video  Recording  of  Experiment 

Please  be  aware  that  we  are  making  video  recording  of  this  baseline  study  for  a  comparison  with 
future  systems.  If  you  strongly  object  to  having  yourself  recorded  as  you  participate  in  this 
simulation,  please  inform  the  experimenters  immediately. 
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6.0  BACKGROUND  QUESTIONNAIRE 


Controller  ID:  abed  Date 

Test  System:  ARTS  IIIA  /  ARTS  IIIE  /  STARS 


Instructions 

The  purpose  of  this  questionnaire  is  to  obtain  information  concerning  your  experience  and  background.  This 
information  will  be  used  to  describe  the  participants  in  this  study  as  a  group.  So  that  your  identity  can  remain 
anonymous,  your  actual  name  should  not  be  written  on  this  form.  Instead,  your  data  will  be  identified  by  a  controller 
code  known  only  to  yourself  and  the  experimenters. 


1)  What  is  your  age? 

_ years 

2)  How  many  years  have  you  actively  controlled  traffic? 

_  years 

3)  How  many  years  have  you  used  the  ARTS  IIIA  system? 

_  years 

4)  How  many  of  the  past  12  months  have  you  actively  controlled  traffic? 

_  months 

5)  What  is  your  current  position  as  an  air  traffic  controller? 

□  Developmental  □  Full  Performance  Level  □  Other  (specify) _ 

6)  In  which  environment  do  you  have  the  most  experience  as  an  air  traffic  controller? 

□  En  Route  □  Terminal  □  Other  (specify) _ 

7)  If  you  wear  corrective  lenses,  will  you  have  them  with  you  to  wear  during  the  simulation? 

Q  Yes  □  No  □  I  don’t  wear  corrective  lenses 

8)  Circle  the  number  which  best  describes  your  current  state  of  health. 

1  2  3  4  5  6  7 

Not  Very 
Healthy 

9)  Circle  the  number  which  best  describes  your  current  skill  as  an  air  traffic  controller. 

1  2  3  4  5  6  7 

Not  Very 
Skilled 

10)  Circle  the  number  which  best  describes  your  level  of  experience  with  personal  computers. 

1  2  3  4  5  6  7 

Not  Very 
Experienced 

11)  Circle  the  number  which  best  describes  your  level  of  satisfaction  with  the  ARTS  IIIA. 

1  2  3  4  5  6  7 

Not  Very 
Satisfied 


8 

Extremely 

Experienced 

8 

Extremely 

Satisfied 


8 

Extremely 

Healthy 

8 

Extremely 

Skilled 
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Appendix  C 

Measure  Summary  and  Sector  Data 


Table  C-I .  Measure  Summary  Table  for  All  Constructs 
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Values  in  parentheses  show  standard  deviations. 


Table  C-1 .  Measure  Summary  Table  for  All  Constructs  (Cont.) 
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8  “Not  Applicable”  indicates  that  it  was  not  appropriate  to  report  an  average  or  sum  across  sectors  for  this  variable. 

9  These  values  are  average  ratings  made  by  the  controllers  on  the  post-run  questionnaires.  The  ratings  ranged  from  1  (strongly  disagree)  to  8  (strongly  agree). 
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Table  C-2,  Quantitative  Sector  Data:  Means  for  Each  Position  in  Each  Configuration 


Fidelity  Number  of  Departures  803  25.1  26.5  1.0  79.2  24.5  26.2 

Note.  D  -  Initial  Departure  Sector,  S  >  South  Sector,  R  -  Rockport  Sector,  and  F  -  Final  One  Sector 


Table  C-3.  Quantitative  Sector  Data:  Standard  Deviation 
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Table  C-4.  Questionnaire  Data:  Mean  Ratings 
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Table  C-6.  ARTS  Mean  Message  Entries  Per  Sector 


Runway  Configuration  27/22L 
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Table  C-7.  Mean  Data  For  4R/L  Runway  Configuration  by  Sector  and  Week 
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Table  C-8.  Sector  by  Week  For  4R/L  Runway  Configuration  Standard  Deviation  Data 
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Variable 

Operational  Errors 

Conflict  Alerts 

Aircraft  Under  Control 

Average  Time  in  Airspace  (min) 

Average  Arrival  Time 

Average  Departure  Time 

Altitude  Assignments  per  Aircraft 

Average  Spacing  on  Final 
Approach  (feet) 

Minutes  Between  Landings 

Data  Entries 

Data  Entry  Errors 

Number  of  Altitude,  Speed,  and 
Heading  Changes 

Workload  per  Aircraft 

Average  Workload 

Post-Run  Workload 

Communication  Workload 

Data  Entry  Workload 

Number  of  Arrivals 

Number  of  Departures 

ATC  Services  (Pilot) 

How  Well  Did  You  Control 

Realism 

Technical  Problems 

Construct 

Safety 

Capacity 

Performance 

Workload 

Simulation 

Fidelity 

Questionnaire 

Problem  Difficulty _ 1.26  1.29  1.50  1.50  1.34  |  1.00  |  1.57  |  1.24  |  1.29  |  1.50  |  0.96  |  0.58 


Table  C-9.  27/22L,  Sector  D  -  15-Minute  Interval  Means 


Construct 

Variable 

1 

2 

3 

4 

5 

6 

Capacity 

Aircraft  Under  Control 

""n.o 

17.3 

21.0 

24.7 

18.1 

20.6 

Altitude  Assignments  per  Aircraft 

1.2 

0.9 

0.9 

0.9 

0.8 

1.2 

Performance 

Data  Entries 

33.3 

33.4 

35.8 

37.8 

36.8 

36.5 

Data  Entry  Errors 

1.6 

0.9 

2.0 

1.0 

1.2 

1.3 

Workload 

Workload  per  Aircraft 

1.2 

0.9 

0.9 

0.9 

1.0 

0.8 

Average  Workload 

2.9 

2.8 

3.2  - 

3.5 

2.8 

2.8 

Communication  Workload 

3.5 

3.1 

3.1 

3.1 

3.2 

4.1 

Data  Entry  Workload 

2.0 

1.9 

1.7 

1.5 

2.0 

1.8 

i.  .  ■ 

Note:  All  values  are  averaged  across  runs. 


Table  C-10.  27/22L,  Sector  S  -  15-Minute  Interval  Means 


Construct 

Variable 

1 

2 

3 

4 

5 

6 

Capacity 

Aircraft  Under  Control 

10.0 

11.5 

11.3 

15.4 

12.6 

18.6 

Altitude  Assignments  per  Aircraft 

0.7 

0.9 

1.1 

1.0 

0.8 

0.6 

Performance 

Data  Entries 

22.3 

21.6 

24.8 

28.2 

27.7 

34.0 

Data  Entry  Errors 

1.8 

0.3 

1.7 

L3 

0.2 

1.3 

Workload 

Workload  per  Aircraft 

0.3 

0.2 

0.2 

0.2 

0.2 

0.2 

Average  Workload 

2.8 

2.1 

El 

3.4 

2.9 

3.7 

Communication  Workload 

7.1 

8.3 

7.1 

5.9 

Data  Entry  Workload 

2.3 

1.9 

2.3 

1.9 

2.3 

1.8 

Table  C-1 1.  27/22L,  Sector  R  -  15-Minute  Interval  Means 


Construct 

Variable 

1 

2 

3 

4 

5 

6 

Capacity 

Aircraft  Under  Control 

15.1 

16.8 

10.0 

18.0 

16.1 

13.5 

Altitude  Assignments  per  Aircraft 

0.9 

1.3 

0.8 

0.7 

1.0 

0.9 

Performance 

Data  Entries 

32.0 

40.4 

32.5 

38.7 

40.0 

39.7 

Data  Entry  Errors 

0.5 

0.7 

0.8 

1.5 

1.7 

1.8 

Workload 

Workload  per  Aircraft 

0.2 

0.3 

0.3 

0.2 

0.2 

0.3 

Average  Workload 

3.5 

4.2 

3.0 

3.7 

3.8 

3.8 

Communication  Workload 

3.3 

El 

3.2 

3.1 

3.9 

3.1 

Data  Entry  Workload 

2.1 

2.5 

3.2 

2.1 

2.5 

3.0 

Table  C-1 2.  27/22L,  Sector  F  -  15-Minute  Interval  Means 


Construct 

Variable 

1 

_ 

2 

3 

4 

5 

6 

Capacity 

Aircraft  Under  Control 

18'4 

14.2 

18.4 

21.0 

16.9 

Altitude  Assignments  per  Aircraft 

1.2 

1.7 

1.4 

1.3 

1.3 

1.5 

Average  Spacing  on  Final  Approach  (feet) 

39,748 

24,070 

31,451 

31,311 

26,561 

28,892 

Minutes  Between  Landings 

2.3 

2.5 

2.9 

3.8 

2.3 

Kn 

Performance 

Data  Entries 

17.6 

19.7 

17.2 

23.0 

24.0 

20.5 

Data  Entry  Errors 

0.8 

0.3 

0.0 

0.2 

0.8 

0.3 

Workload 

Workload  per  Aircraft 

0.2 

0.2 

0.2 

0.2 

0.2 

0.3 

Average  Workload 

2.2 

4.1 

3.1 

3.9 

4.6 

4.3 

Communication  Workload 

10.0 

9.3 

8.0 

8.4 

8.0 

8.1 

Data  Entry  Workload 

2.0 

1.0 

1.2 

1.2 

1.1 

1.2 

Table  C-13.  4R/L,  Sector  D  -  15-Minute  Interval  Means 


Construct 

Variable 

1 

2 

3 

4 

5 

6 

Capacity 

Aircraft  Under  Control 

17.0 

16.5 

21.1 

23.5 

17.6 

19.5 

Altitude  Assignments  per  Aircraft 

1.1 

0.9 

1.0 

0.9 

1.0 

0.8 

Performance 

Data  Entries 

33.1 

31.4 

34.8 

40.3 

38.4 

31.1 

Data  Entry  Errors 

1.3 

0.3 

0.7 

1.7 

1.0 

0.8 

Workload 

Workload  per  Aircraft 

0.2 

0.2 

0.1 

0.1 

0.2 

0.2 

Average  Workload 

2.9 

2.7 

2.8 

3.1 

3.1 

3.6 

Communication  Workload 

MM 

3.1 

2.8 

wgm 

3.2 

6.1 

Data  Entry  Workload 

1.9 

ON 

1.6 

1.7 

2.1 

1.6 

Table  C-14.  4R/L,  Sector  S  -  15-Minute  Interval  Means 


Construct 

Variable 

1 

2 

3 

4 

5 

6 

Capacity 

Aircraft  Under  Control 

9.7 

9.2 

10.5 

13.9 

10.6 

15.5 

Altitude  Assignments  per  Aircraft 

0.4 

0.7 

0.7 

0.6 

0.5 

0.6 

Performance 

Data  Entries 

20.0 

19.0 

24.0 

22.1 

20.4 

22.9 

Data  Entry  Errors 

0.2 

0.2 

0.8 

0.3 

0.8 

1.0 

Workload 

Workload  per  Aircraft 

0.2 

0.3 

0.2 

0.2 

0.2 

0.2 

Average  Workload 

2.3 

2.3 

2.4 

2.9 

2.5 

4.0 

Communication  Workload 

8.6 

7.7 

8.3 

7.3 

7.6 

4.3 

Data  Entry  Workload 

2.1 

2.0 

2.3 

1.6 

2.0 

1.5 

Table  C-15.  4R/L,  Sector  R  -  15-Minute  Interval  Means 


Construct 

Variable 

1 

2 

3 

4 

5 

6 

Capacity 

Aircraft  Under  Control 

15.0 

19.4 

8.7 

17.8  ” 

17.5 

16.0 

Altitude  Assignments  per  Aircraft 

0.9 

1.2 

0.9 

0.7 

0.9 

0.9 

Performance 

Data  Entries 

25.0 

32.3 

26.3 

34.4 

31.3 

26.4 

Data  Entry  Errors 

0.5 

0.4 

0.6 

0.6 

0.5 

0.4 

Workload 

Workload  per  Aircraft 

0.2 

0.2 

0.2 

0.2 

0.2 

0.3 

Average  Workload 

3.2 

4.0 

1.8 

3.0 

4.1 

4.5 

Communication  Workload 

52.8 

86.5 

25.6 

51.5 

74.3 

56.2 

Data  Entry  Workload 

1.7 

1.7 

3.1 

1.9 

1.8 

1.7 

Table  C-16.  4R/L,  Sector  F  -  15-Minute  Interval  Means 


Construct 

Variable 

1 

2 

3 

4 

5 

6 

Capacity 

Aircraft  Under  Control 

7.6 

20.3 

14.7 

15.4 

18.6 

17.2 

Altitude  Assignments  per  Aircraft 

1.4 

2.4 

2.6 

2.0 

1.8 

1.6 

Average  Spacing  on  Final  Approach  (feet) 

30,663 

28,096 

24,334 

27,878 

19,836 

21,012 

Minutes  between  landings 

0.0 

1.9 

2.8 

UJH 

2.1 

1.7 

Performance 

Data  Entries 

15.0 

17.6 

17.8 

19.4 

14.8 

14.9 

Data  Entry  Errors 

0.9 

0.1 

0.1 

0.0 

0.0 

0.1 

Workload 

Workload  per  Aircraft 

0.3 

0.2 

0.2 

! 

0.2 

0.2 

■a 

Average  Workload 

2.4 

3.7 

3.0 

3.2 

4.1 

4.1 

Communication  Workload 

11.0 

8.7 

7.1  1 

1 

8.6 

9.0 

7.0 

Data  Entry  Workload 

2.0 

0.9 

1.3 

1.3 

0.8 

0.9 

C-11 


Table  C-17.  27/22L,  Sector  D  -  15-Minute  Interval  Standard  Deviation  Means 


Construct 

Variable 

1 

2 

3 

4 

5 

6 

Capacity 

Aircraft  Under  Control 

0.00 

0.95 

0.94 

0.82 

0.88 

1.35 

Altitude  Assignments  per  Aircraft 

0.17 

0.14 

0.15 

0.09 

0.15 

0.17 

Performance 

Data  Entries 

18.05 

11.18 

6.85 

5.31 

5.78 

5.21 

Data  Entry  Errors 

1.85 

1.07 

1.67 

1.26 

1.60 

0.82 

Workload 

Workload  per  Aircraft 

0.17 

0.14 

0.15 

0.09 

0.11 

0.15 

Average  Workload 

1.48 

1.64 

1.63 

1.97 

1.54 

1.73 

Communication  Workload 

0.59 

0.51 

0.41 

0.53 

0.63 

2.32 

Data  Entry  Workload 

1.06 

0.66 

0.35 

0.25 

0.25 

0.31 

Table  C-18.  IIUTL,  Sector  S  -  15-Minute  Interval  Standard  Deviation  Means 


Construct 

Variable 

1 

2 

3 

4 

5 

6 

Capacity 

Aircraft  Under  Control 

0.47 

0.71 

1.16 

0.84 

0.97 

1.17 

Altitude  Assignments  per  Aircraft 

0.38 

0.59 

0.44 

0.28 

0.46 

0.18 

Performance 

Data  Entries 

11.31 

11.47 

7.14 

5.23 

11.91 

9.01 

Data  Entry  Errors 

1.83 

0.49 

1.51 

1.51 

0.41 

1.63 

Workload 

Workload  per  Aircraft 

0.14 

0.12 

0.14 

0.11 

0.12 

0.09 

Average  Workload 

1.36 

1.31 

1.45 

1.56 

1.44 

1.65 

Communication  Workload 

0.94 

0.69 

0.82 

0.93 

1.24 

0.69 

Data  Entry  Workload 

1.14 

1.07 

0.75 

0.35 

1.03 

0.54 

Table  C-19.  27/22L,  Sector  R  -  15-Minute  Interval  Standard  Deviation  Means 


Construct 

Variable 

1 

2 

3 

4 

5 

6 

Capacity 

Aircraft  Under  Control 

0.32 

1.32 

0.47 

0.67 

2.02 

0.97 

Altitude  Assignments  per  Aircraft 

0.21 

0.20 

0.26 

0.16 

0.20 

0.24 

Performance 

Data  Entries 

15.81 

12.58 

10.13 

5.92 

2.97 

13.78 

Data  Entry  Errors 

0.76 

0.76 

1.17 

1.38 

0.82 

2.23 

Workload 

Workload  per  Aircraft 

0.12 

0.13 

0.17 

0.08 

0.10 

0.13 

Average  Workload 

1.78 

2.01 

1.58 

1.48 

1.79 

1.59 

Communication  Workload 

1.28 

0.58 

0.76 

0.69 

0.66 

0.69 

Data  Entry  Workload 

1.03 

0.85 

0.98 

0.27 

0.39 

0.95 

Table  C-20,  27/22L,  Sector  F  -  15-Minute  Interval  Standard  Deviation  Means 


Construct 

Variable 

1 

2 

3 

4 

5 

6 

Capacity 

Aircraft  Under  Control 

0.95 

1.07 

0.79 

2.17 

1.33 

2.23 

Altitude  Assignments  per  Aircraft 

0.43 

0.58 

0.44 

0.41 

0.51 

0.57 

Average  Spacing  on  Final  Approach 

6,181 

8,254 

14,443 

16,111 

13,052 

13,334 

Minutes  between  landings 

0.83 

1.46 

1.59  ■ 

2.83 

1.31 

1.53 

Performance 

Data  Entries 

7.89 

6.82 

5.27 

5.29 

3.52 

3.08 

Data  Entry  Errors 

1.16 

0.49 

0.00 

0.41 

0.98 

0.52 

Workload 

Workload  per  Aircraft 

0.11 

0.09 

0.12 

0.10 

0.09 

0.09 

Average  Workload 

0.98 

1.67 

1.62 

1.74 

1.95 

1.35 

Communication  Workload 

1.65 

0.96 

1.20 

1.04  ' 

0.89 

1.31 

Data  Entry  Workload 

0.87 

0.35 

0.38 

0.26 

0.18 

0.22 

C-12 


Table  C-21.  4R/L,  Sector  D  -  15-Minute  Interval  Standard  Deviation  Means 


Construct 

Variable 

1 

2 

3 

4 

5 

6 

Capacity 

Aircraft  Under  Control 

0.00 

0.97 

0.28 

0.78 

0.96 

2.07 

Altitude  Assignments  per  Aircraft 

0.12 

0.15 

0.08 

0.11 

0.14 

0.13 

Performance 

Data  Entries 

9.61 

19.37 

10.49 

17.56 

17.87 

14.52 

Data  Entry  Errors 

1.35 

0.50 

1.32 

1.94 

1.41 

1.16 

Workload 

Workload  per  Aircraft 

0.07 

0.08 

0.07 

0.06 

0.09 

0.10 

Average  Workload 

1.23 

1.28 

1.43 

1.46 

1.51 

2.13 

Communication  Workload 

0.30 

0.46 

0.35 

0.37 

0.44 

3.24 

Data  Entry  Workload 

0.57 

1.24 

0.50 

0.73 

0.93 

0.79  ^ 

Table  C-22.  4R/L,  Sector  S  -  15-Minute  Interval  Standard  Deviation  Means 


Construct 

Variable 

1 

2 

3 

4 

5 

6 

Capacity 

Aircraft  Under  Control 

0.48 

1.09 

0.66 

0.64 

0.87 

1.81 

Altitude  Assignments  per  Aircraft 

0.17 

0.24 

0.15 

0.20 

0.14 

0.18 

Performance 

Data  Entries 

7.84 

13.82 

8.34 

4.99 

3.07 

8.49 

Data  Entry  Errors 

0.60 

0.44 

0.83 

0.71 

0.89 

1.69 

Workload 

Workload  per  Aircraft 

0.13 

0.17 

0.15 

0.12 

0.13 

0.13 

Average  Workload 

1.31 

1.47 

1.44 

1.67 

1.36 

2.15 

Communication  Workload 

0.59 

0.93 

0.61 

1.09 

1.19 

1.96 

Data  Entry  Workload 

0.80 

1.21 

0.86 

0.31 

0.36 

0.58 

Table  C-23.  4R/L,  Sector  R  -  15-Minute  Interval  Standard  Deviation  Means 


Construct 

Variable 

1 

2 

3 

4 

5 

6 

Capacity 

Aircraft  Under  Control 

0.41 

0.96 

0.63 

0.44 

1.05 

1.53 

Altitude  Assignments  per  Aircraft 

0.20 

0.24 

0.18 

0.11 

0.11 

0.27 

Performance 

Data  Entries 

11.78 

13.97 

10.58 

7.37 

8.61 

10.60 

Data  Entry  Errors 

0.82 

0.53 

0.88 

0.88 

0.76 

0.52 

Workload 

Workload  per  Aircraft 

0.10 

0.06 

0.13 

0.08 

0.10 

0.12 

Average  Workload 

1.54 

1.29 

0.98 

1.39 

1.80 

1.87 

Communication  Workload 

0.62 

0.32 

0.35 

0.32 

0.57 

1.23 

Data  Entry  Workload 

0.79 

0.73 

1.38 

0.41 

0.59 

0.72 

Table  C-24.  4R/L,  Sector  F  -  15-Minute  Interval  Standard  Deviation  Means 


Construct 

Variable 

1 

2 

3 

4 

5 

6 

Capacity 

Aircraft  Under  Control 

0.77 

1.18 

1.25 

1.50 

1.61 

2.28 

Altitude  Assignments  per  Aircraft 

0.37 

0.47 

0.52 

0.41 

0.59 

0.46 

Average  Spacing  on  Final  Approach 

5,878 

15,254 

16,614 

15,343 

9,576 

11,122 

Minutes  between  landings 

0.00 

1.23 

2.38 

2.80 

1.84 

1.78 

Performance 

Data  Entries 

8.39 

6.11 

4.41 

8.73 

5.63 

8.53 

Data  Entry  Errors 

2.70 

0.33 

0.33 

0.00 

0.00 

0.35 

Workload 

Workload  per  Aircraft 

0.17 

0.07 

0.12 

0.11 

0.10 

0.08 

Average  Workload 

1.14 

1.52 

1.63 

1.68 

1.70 

1.55 

Communication  Workload 

1.71 

0.71 

1.04 

1.65 

1.54 

1.51 

Data  Entry  Workload 

1.14 

0.31 

0.35 

0.62 

0.27 

0.49 

CA3 


Appendix  D 
Controller  Comments 


The  following  data  represent  controller  responses  (edited  for  grammar)  to  Sections  D.l,  D.2,  and 
E  of  Final  Questionnaire.  Responses  for  Section  D.l  focus  on  improving  specific  components  of 
the  ARTS  DIA  console.  Responses  for  Section  D.2  consist  of  mistakes  controllers  commonly 
made  using  the  ARTS  DIA  console  and  potential  causes  of  these  mistakes.  Responses  on  Section 
E  concern  the  baselining  effort. 

WEEK  1 


Controller  Responses  to  Section  D.  1 

The  scopes  lack  consistency  in  all  areas  of  radar  and  alphanumeric  function  displays.  Eliminate 
existing  trackball  and  keyboard,  replace  with  mouse,  and  keyboard  with  windows.  All  tied  into 
NAS.  Get  better  weather  displays.  An  interface  between  the  ARTS  tag  and  the  NAS  would  be 
helpful  (i.e.,  when  making  an  entry  on  an  ARTS  tag  such  as  altitude,  aircraft  type,  could  be 
interfaced  to  eliminate  the  FDIO  data  entry). 

Radar  maps  could  be  sharper.  This  is  the  only  equipment  I’ve  worked  with.  I’ve  learned, 
through  the  years,  to  become  efficient  in  the  way  I  interact  with  the  equipment.  I  am  working 
with  simulation  equipment  that  has  a  radar  display  using  a  windows-type  program.  It  is  very  easy 
to  use,  it  allows  the  controller  to  tailor  the  position  to  his/her  liking.  It  has  been  my  experience 
that  a  mouse  is  not  as  effective  as  a  trackball.  The  mouse  isn’t  stationary  and  easily  misplaced. 
Also  a  mouse  is  not  as  durable  as  a  trackball. 

Map  displays  should  be  more  precise.  We  are  supposed  to  stay  1.5  miles  from  a  boundary; 
however,  there  are  times  the  boundary  line  is  1  mile  wide  itself.  A  finer  line  would  reduce  error. 
Keyboard:  The  current  configuration  is  not  user  friendly.  A  keyboard  more  like  a  computer 
keyboard  would  be  easier.  Trackball:  If  we  had  a  system  where  you  could  touch  the  screen  for  a 
handoff,  etc.,  then  you  wouldn’t  need  a  trackball.  ARTS  characters:  Right  now,  there  are  four 
preset  sizes  and  only  two  are  even  close  to  being  usable.  A  better  method  changing  character 
size  would  be  good. 

We  are  limited  to  what  we  can  enter  into  the  keyboard  because  of  the  programs  (i.e.,  we  are 
unable  to  enter  an  IFR  flight  plan  into  the  NAS  system  using  our  keyboards).  Center  can  do  this 
function.  It  would  cut  down  on  our  workload  if  we  were  able  to  accomplish  this  in  the  terminal 
environment. 

Supervisor  Response  to  Section  D.  1 

I  am  an  avid  fan  of  a  windows-based  system  with  feature  such  as  pull-down  menus,  multi-tasking 
(window  in  a  window),  mouse  applications  vs.  slow  trackball.  Maps  need  to  be  digitized  and 
displays  enlarged  to  not  only  be  more  useful  but  provide  additional  working  space  at  the  console. 
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Controller  Responses  to  Section  D.2 


Some  keyboard  commands  are  quite  lengthy  and  when  traffic  builds  up  it’s  easy  to  mis-hit  the 
keys.. .maybe  the  keyboard  is  too  small? 

Sometimes  people  will  ship  the  aircraft  to  the  next  sector  thinking  the  handoff  has  been 
accomplished  because  of  his  position  and  not  his  data  block.  Maybe  a  color  change  would  show 
a  handoff. 

Overall  the  keyboards  work  well  but  when  they  start  to  stick,  it  creates  much  more  workload  on 
the  controller. 

Supervisor  Response  to  Section  D.2 

In  the  environment  that  currently  exists,  you  constantly  fumble  for  the  knobs.  A  system  that  is 
more  user-friendly  for  adjusting  scope  presentation  would  enhance  the  system  tremendously. 
More  room  is  needed  for  work  space  in  front  of  the  PVD. 

Controller  Response  to  Section  E 

Shorten  problems  to  one  hour.  Increase  the  traffic  volume  to  final  and  add  a  second  final 
controller. 

Supervisor  Response  to  Section  E 

To  us,  the  ARTS  IHA  console  is  “home.”  If  you  were  to  replace  our  consoles  at  Boston  with 
ARTS  DIE  equipment,  it  would  probably  excite  everyone!  My  point  is,  design  a  system  that  will 
make  all  users  (HA,  DIA,  HIE)  excited  with  the  change.  Let’s  take  advantage  of  what  we  have 
seen  with  20  in  screens  and  color  presentations  and  integrate  these  products  into  the  replacement 
cycle  for  “all”  systems,  regardless  of  what  currently  exists.  I  know  that  the  intent  in  development 
is  heading  in  this  direction  but  it  is  crucial  to  continue  to  emphasize  this  point.  If  we  are 
baselining,  let’s  establish  the  criteria  for  today’s  technology  as  a  start  and  continue  to  build  from 
there.  The  foundation  needs  to  be  technology  from  199-now! 
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Controller  Responses  to  Section  D.  1 

Reliability  of  ASR-9  radar;  Reliability  of  ARTS:  Less  system  crashes  or  scatters,  no  false  targets, 
no  software  problems.  Setup  -  automatic,  personalized  display  brightness  setup  via  computer 
card  or  access  code. 

There  is  too  much  glare  in  the  glass.  Not  enough  room  to  write  if  needed-trackball  and  keypack 
get  in  the  way.  The  console  knobs  can  be  difficult  to  identify. 

Radar  maps  and  display:  Digital  display  would  be  much  better  than  analog.  Keyboard  entries 
should  be  integrated  into  the  NAS  and  FDIO.  The  ARTS  DIA  interface  should  be  more  user- 
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friendly.  The  scope  set  procedures  are  cumbersome,  and  it  would  be  nice  if  it  could  be 
automated. 

r 

A  larger  workspace  would  be  nice.  Trackballs  frequently  fail  or  work  improperly  causing 
controller  stress  to  rise.  I  realize  ARTCC  and  terminal  duties  are  different-however,  being  a 
former  en  route  controller,  I  think  the  NAS  PVDs  are  much  more  efficient  and  user-friendly  than 
ARTS  TTTA  equipment  is.  Additionally,  my  experience  has  been  that  the  NAS  equipment  is  much 
more  reliable  than  ARTS  IIIA  equipment. 

Supervisor  Response  to  Section  D.l 

Map  displays  should  allow  labeling  of  airways,  routes,  fixes,  blocks  of  airspace,  altitude  stratums 
(not  that  all  of  these  would  be  used  simultaneously,  but  at  ATC’s  preference)  in  subdued  colors. 
Alphanumerics  should  have  capability  to  be  enlarged  or  reduced  with  set  sizes.  WX  should  have 
color  capability.  Primary  and  beacon  returns  should  have  different  shades  of  color.  Controls 
should  be  grouped  by  similar  function  (i.e.,  beacon  and  primary  together,  display  intensity  and 
adjustments  together).  Work  areas  should  have  non-equipment-cluttered  writing  areas. 

Keyboards  should  be  close  to  QWERTY  w/F  keys. 

Controller  Responses  to  Section  D.2 

Slightly  missing  targets  or  keyboard  alphanumeric  keys. 

Keyboard  often  sticks  and  it  can  be  difficult  to  find  the  preview  area  among  the  alphanumeric. 

The  slewball  has  to  be  almost  right  on  the  headset  in  order  to  have  an  effect  i.e.,  difficult  during 
heavy  traffic. 

Keyboard  often  goes  haywire  with  random  and/or  rogue  entries  appearing  without  controller 
input.  Addition2illy  keyboard  entries  are  often  cumbersome  and/or  lengthy  which  causes  me  to 
divert  my  attention  from  traffic  control  duties.  Often  have  difficulty  in  distinguishing  and 
selecting  correct  data  blocks  with  trackball(s). 

Supervisor  Response  to  Section  D.2 

Alphanumerics  are  difficult  to  read,  especially  since  letter  exit  fixes  shared  with  altitude 
information.  M350  is  turned  N-BND,  N350  turned  S-BND,  sometimes  misread  aircraft  is 
turned  wrong  way.  Know  adjustments  are  difficult  in  the  dark  environment,  often  involve 
guessing  which  knob,  watching  what  happens  when  you  turn  it,  and  trying  another  guess. 
Keyboard  entries  FDIO  are  QWERTY,  ARTS  is  alphabetized  creating  hunt  and  peck.  Many 
format  errors  because  of  vertical  display  of  entry  information,  makes  spacing  functions  hard  to 
detect.  FP  changes  should  be  accomplished  through  radar  console,  not  by  having  to  move  over  to 
FDIO  and  changes.  Complete  flight  plan  information  should  be  displayed  at  position  with  edit 
capability  (windows). 

Controller  Responses  to  Section  E 

[Sim]  Pilots  and  software  used  to  help  them  control  traffic  need  improvement. 
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The  work  we  did  with  simulation  was  good  but  frustrating.  The  realism  could  be  improved  by 
better  aircraft  compliance. 

As  far  as  this  study  goes,  every  attempt  at  realism  must  be  achieved  or  attempted.  Current  active 
controllers  must  feel  challenged  and  feel  like  the  simulation  is  real  to  get  maximum  participation 
out  of  the  scenario.  Sim  pilots  and  sim  pilot  software  must  achieve  more  operational  consistency 
to  achieve  better  realism;  try  and  make/request  sim  pilots  to  become  more  aware  of  aviation/ATC 
phraseology  again  in  an  effort  to  promote  realism.  Can  we  have  more  scenarios  (i.e.,  other 
runway  configurations  33/27, 22  just  to  prevent  complacency  from  controller  study  group)? 

Supervisor  Response  to  Section  E 

Perhaps  realism  should  be  explained  to  the  team.  What  you  may  consider  realistic  could  be  very 
different  from  the  sensitivity  of  realism  the  controllers  have.  What  you  need  for  study  purposes 
probably  is  not  as  detailed  as  what  controllers  may  be  expecting  for  realism  and,  if  this  is 
explained  to  them,  they  may  not  be  as  frustrated  when  things  get  silly.  Make  sure  there  are  no 
surprises.  Brief  talks  on  problems  with  aircraft  compliance  of  ATC  instructions  whether  it’s 
software,  pilots,  etc.  They  should  know  crazy  turns  could  happen,  don’t  get  frustrated,  hang  with 
it,  it’s  not  a  reflection  on  ability.  Supervisor/SME  should  know  their  role  involved  SME 
evaluation  of  controller’s,  logging  problem  events,  and  acting  as  a  TRACON  supervisor, 
sometimes  all  at  the  same  time.  Visit  to  SIMOPs  would  help  ATCs  understand  the  equipment 
and  limitations  of  the  pilots-reduce  frustration. 

WEEK  3 


Controller  Responses  to  Section  D.  1 

Radar  and  Map  displays  should  be  sharper  and  clearer  and  possibly  color  to  display  “shelving” 
more  readily.  Keyboard  lights  are  constantly  burning  out  or  too  bright  compared  to  other  buttons 
alongside.  Too  many  entries  are  required  for  seemingly  simple  operations:  multi-function  key, 
green  keys,  and  so  on.  Console  switches  and  knobs:  just  plain  old  and  outdated.  Basically, 
functions  should  be  able  to  quickly  and  readily  let  a  controller  make  an  entry  so  his  eyes  can  go 
back  to  the  radar  screen  sooner.  Possibly  and  voice-activated-system  of  recognizing  what  aircraft 
you’re  talking  to  and  being  able  to  enter  data  by  just  saying  it  (i.e.,  handoff  or  call  sign  being 
entered  just  by  speaking  it,  for  VFR  pop-ups).  This  would  always  let  the  controller  keep  his  eyes 
on  the  screen.  A  big  plus!  Thank  you! 

Workspace:  console  is  too  narrow  to  write  on  a  normal  8  1/2x11  paper.  Keyboard:  not 
typewriter  oriented  therefore  limits  workspeed.  Switches:  mechanical  and  worn,  resulting  in 
rough  movement.  Trackball:  Sticky,  no  regular  movement.  Radar:  full  data  displays  of  flight 
plans  should  be  available  on  the  scope,  as  well  as  the  capability  to  amend  that  information  at  the 
scope.  Other:  Real  time  data  on  other  monitors  should  be  available  in  a  windows  format,  such 
as  wind,  altimeter,  and  weather.  Now  that  information  is  placed  in  three  different  locations.  A 
display  that  could  operate  lighted  conditions  would  be  beneficial.  The  first  few  minutes  of  each 
controller’s  session  on  a  particular  position  is  spent  setting  it  up  the  way  he/she  likes  it.  It  would 
seem  to  free  up  some  scanning  time  if  the  new  equipment  had  a  programmable  memory  of  each 
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controller’s  desired  setting.  Extra  voice  coordination  could  be  eliminated  via  a  message  sending 
more  from  screen  to  screen. 

The  maps  on  all  ARTS  IHA  have  not  been  what  I’d  like  to  see.  They  are  usually  too  wide,  out  of 
alignment,  and  out  of  focus.  I’m  left-handed,  as  far...keypack  position  needs  to  be  identified. 

The  FDIO/Host  computer  should  be  able  to  be  connected  into  ARTS— that  way  amendments, 
flight  plan  information,  WX  information,  is  all  right  at  your  position. 

The  radar  and  map  displays  tend  to  usually  become  washed  out  or  become  enlarged  to  the  point 
where  it  takes  your  attention  away  from  your  primary  duties.  We  need  a  system  that  will  provide 
radar  coverage  from  the  surface,  and  that  will  not  be  affected  from  obstruction  or  terrain.  The 
keyboard  would  be  more  user-friendly  if  it  was  moveable  to  allow  for  personal  comfort.  The 
console  and  switches  are  usually  either  too  touchy  where  they  become  very  hard  to  use  or  they 
don’t  work  as  they  should,  causing  the  controller  to  sometimes  not  get  the  best  display  possible 
(i.e.,  weather  radar).  The  keypack  should  be  moveable  -  the  display  should  be  of  a  brighter  and 
have  more  contrast  than  now.  The  video  maps  should  be  more  constant  (less  blooming  and 
thinner  lines.)  You  should  also  be  able  to  look  at  flight  plans  on  radar  console  and  make  changes 
to  the  flight  plans  and  the  radar  console.  This  would  reduce  workload  because  the  need  for 
coordinating  these  changes  would  not  be  necessary.  In  addition,  the  interface  between  facilities 
could  be  improved. 

Supervisor  Response  to  Section  D.  1 

Radar  and  map  displays  are  never  exact.  They  are  often  blurred,  washed  out,  and  usually 
misaligned.  Keyboard  hardware  and  trackball  hardware  are  always  in  need  of  repair/adjustment 
or  replacement.  The  ease  of  using  the  trackballs  varies  with  each  position.  Many  of  our  ARTS 
entries  have  become  too  lengthy.  They  are  difficult  to  teach  because  there  are  too  many  of  them. 
These  entries  also  do  not  allow  for  minor  errors  such  as  a  space  or  character  too  many.  A  bad 
entry  is  not  easily  corrected  and  must  usually  be  completely  re-entered.  Moving  the  preview  and 
systems  areas  and  the  various  tab  lists  should  be  a  click  and  drag  function  that  does  not  require  a 
keyboard  entry.  Keyboard  should  be  adaptable  for  left-handed  individuals. 

Controller  Responses  to  Section  D.2 

The  amount  of  keys  to  hit  when  making  various  entries  requires  attention  to  be  diverted  to 
looking  at  keyboard  when  you  need  to  be  constantly  watching  scope. 

Entries  -  selecting  wrong  keys.  Trackball  -  hard  to  discern  to  the  slew  overlapping  targets. 
Switches  and  knobs  -  decentering  the  presentation  is  tricky  because  it  is  touchy.  You  must  very 
gently  turn  the  knob  in  order  to  avoid  the  ‘picture’  going  off  the  scope.  But  sometimes  it  still 
happens. 

Not  knowing  why  many  times  the  ARTS  information/tag  doesn’t  auto  acquire  when  all  of  the 
correct  input  is  there  for  no  rhyme  or  reason.  When  changing  flight  plan  information  to  get  the 
ARTS  TTTA  to  coincide  with  the  FDIO/host  computer  is  much  more  difficult  than  it  should  be. 
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On  a  day-to-day  basis,  we  are  forced  to  use  several  different  functions  that  are  time  consuming, 
and  make  controller  take  eyes  off  the  radarscope.  The  trackballs  are  not  always  easy  to  use;  they 
sometimes  stick. 

Supervisor  Response  to  Section  D.2 

Errors  in  entries  are  not  always  easy  to  recognize.  Depending  upon  where  the  ARTS  preview 
area  is,  a  controller  might  make  several  entries  before  realizing  that  none  of  them  was  accepted 
because  of  the  first  bad  entry.  Often  the  tab  lists,  systems  areas,  preview  areas,  etc.  will  obscure 
aircraft  targets  and  ARTS  tag  information. 
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Supervisor  Responses 

DT.  Digital  displays  with  color  could/would  provide  a  more  effective  way  of  displaying  data  of 
varying  types  (i.e.,  weather,  maps,  data  blocks).  The  current  volume  of  workspace  is  inhibitive  to 
complete  necessary  forms  or  tasks  as  required  nationally  or  locally  (i.e.,  PIREP  forms,  sign  in/out 
forms).  Current  switches  and  knobs  are  not  properly  labeled  as  a  result  of  function  changes  with 
new  ASR9  systems  and  are  quite  cumbersome  to  operate  smoothly  (particularly  decentering 
displays).  ARTS  IHA  interfacing  within  the  NAS  system  is  generally  misunderstood  by 
controllers.  We  would  like  to  have  the  functionality/capability  to  effectively  cause  a  change  in 
the  data  block  transferred  to  the  actual  flightplan  rather  than  duplicating  some  efforts  through 
FDIO/FDEP  equipment.  Calling  up  information  such  as  provided  by  FDIO/FDEP  at  the  radar 
position  would  be  a  welcomed  addition.  The  keyboard  is  cumbersome,  fails  to  follow  keyboard 
standards,  and  results  in  spending  too  much  time  looking  away  from  the  radar  display  while 
entering  information.  A  more  intuitive  interface  may  reduce/eliminate  the  keyboard  for 
controllers  and  use  a  built-in  system  within  the  display.  Maps  and  their  clarity  of  display  on  a 
PVD  would/could  be  sharp/well-defined  in  a  digital  format,  also  permitting  real-time  editing  for 
the  local  facility.  I  would,  overall,  like  my  position/display  to  give  me  anything  that  the  NAS  has 
to  offer  with  regards  to  expected  traffic  loads,  full  flight  plans,  weather  data,  and  administrative 
data  (i.e.,  sign  on/off  currency  tracking).  Perhaps  diagnostics  can  be  enhanced  and  reliability  of 
using  equipment  that  is  proven  sound  (off-the-shelf)  raised. 

D.2.  I  think  lighting  is  a  great  deal  of  concern  when  trying  to  quickly  identify  the  correct  key 
and/or  adjustment  knob  or  switch.  Alphanumerics  on  radar  displays  are  poor  in  resolution  and 
readability  of  an  “S”  to  a  “5”  under  a  quick  scan  can  result  in  misreading  a  call  sign  or  data' 
information. 
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